Differences

This shows you the differences between two versions of the page.

--- admin:grid:creation:oacomp [2009/11/13 17:06]
els
+++ admin:grid:creation:oacomp [2012/10/26 14:27] (current)
@@ Line 1: / Line 1: @@
-====== Creating an Omnidex Grid using Environment Rules ======
+<html><div align="center"><span style="color:red">DRAFT</span></div></html>
+====== Creating an Omnidex Grid using OACOMP ======
-:!: //These instructions show how to create Omnidex Grids using environment files that are compiled with OACOMP.  This is an approach that pre-dated the use of SQL statements such as CREATE DATABASE or CREATE TABLE.  Environment files that are compiled with OACOMP are still supported for backward compatability.//
+:!: //These instructions show how to create Omnidex Grids using environment files that are compiled with OACOMP.  This is an approach that pre-dated the use of SQL statements such as CREATE DATABASE or CREATE TABLE.  Environment files that are compiled with OACOMP are still supported for backward compatability.  These instructions are for Omnidex Version 5.1 and later.  Click [[admin:grid:creation:rules|here]] for instructions for Omnidex Version 5.0.//
+----
+\\
 Omnidex Grids are straightforward to administer.  Each grid node is a separate Omnidex environment, with an environment file, indexes and data.  For convenience, there are methods to share a single Omnidex environment file across multiple nodes.  There are also methods to share data files across multiple nodes.
@@ Line 11: / Line 14: @@
 These instructions describe how to create an Omnidex Grid using Environment Rules.  This was the original method to create Omnidex Grids.  This method has now been replace by use of the [[admin:grid:creation:odxadmin| Omnidex Administrator]] or [[admin:grid:creation:sql | SQL statements]] such as the CREATE DATABASE and CREATE TABLE.
-===== Steps to create an Omnidex Grid using Environment Rules =====
+===== Steps to create an Omnidex Grid using OACOMP =====
 ==== 1. Create an environment pointing to the original database ====
@@ Line 36: / Line 39: @@
 ==== 3.	Distribute both the partitioned and partitioned data across the grid.  ====
-If the database will be logically partitioned, then make sure each grid server has access to the database.  If the database will be physically partitioned, then distribute both the partitioned and unpartitioned data to the grid nodes.  If each node will be in its own directory, then the data files can be copied or symbolically linked to the correct locations.  It is also possible for multiple nodes to share the same physical directories as long as the partitioned data can be differentiated.  To assist this, it is possible to use the environment variable “OMNIDEX_NODE” in the physical clauses for the table, as well as the index prefix for the database.  The variable $OMNIDEX_NODE will be replaced by the node name.
+If the database will be logically partitioned, then make sure each grid server has access to the database.  If the database will be physically partitioned, then distribute both the partitioned and unpartitioned data to the grid nodes.  If each node will be in its own directory, then the data files can be copied or symbolically linked to the correct locations.  It is also possible for multiple nodes to share the same physical directories as long as the partitioned data can be differentiated.
-An example of an environment variable used in a DATABASE … INDEXPREFIX declaration is:
+==== 4. Create the environment file for the Omnidex Grid. ====
-  database       "list"
+The environment file for the Omnidex Grid is a normal environment file; however, it has additional information that describes the locations of the nodes and the partitioning scheme.  There are changes that need to be made for the ENVIRONMENT statement, the DATABASE statement and the TABLE statements.  Note that older versions of Omnidex Grids had a RULE statement at the bottom, and that is no longer necessary.
-    type         flatfile
-    indexprefix  "{idx/$OMNIDEX_NODE/LIST_}"
-An example of a partitioned table is:
+=== The ENVIRONMENT statement ===
-  table          "LIST"
+The ENVIRONMENT statement is expanded to declare the grid nodes and their connection information.  The NODE declarations in the ENVIRONMENT statement are show below.  Consult the documentation on the [[admin:environments:create:oacomp:syntax#environment_statement | ENVIRONMENT statement]] for the complete syntax.
-    physical     "{dat/$OMNIDEX_NODE/list_*.dat}"
-==== 4. Install and build Omnidex indexes on each node. ====
+<code>
+ENVIRONMENT environment
+  [NODE node]
+     [<PARTITIONED | UNPARTITIONED>]
+     [HOST "host"]
+     [PORT n]
+     [ENVIRONMENT "filename"]
+     [OPTIONS "options"]
+</code>
-It is recommended, though not required, that each node be indexed the same.  This reduces the likelihood of queries taking too long because of a lack of indexing on one node.
+Here is an example of an ENVIRONMENT declaration on an Omnidex Grid.
-Indexes can be built in parallel.  For maximum performance, it is recommended that the temporary directory, indicated by the TMPDIR environment variable, be different between builds whenever possible.
+<code>
+environment  "list_env"
+  maxthreads  4
+  node NODE01 partitioned
+  node NODE02 partitioned
+  node NODE03 partitioned
+  node NODE04 partitioned
+</code>
-==== 5. Create an environment file for the grid controller. ====
+=== The DATABASE statement ===
-The environment file for the grid controller has the same database and table layout as the grid nodes.  The physical clauses for the tables can be omitted on the grid controller since the controller only access the tables through the nodes.
+The DATABASE statement is expanded to show the database characteristics for each node. The NODE declarations in the DATABASE statement are show below.  Consult the documentation on the [[admin:environments:create:oacomp:syntax#database_statement | DATABASE statement]] for the complete syntax.
-At the bottom of the environment file, include a rule that describes the Omnidex Grid.  This rule declares the nodes and partitioning scheme used in the Omnidex grid, as well as naming the columns that are guaranteed to have distinct values on each node (discussed earlier).
+<code>
+[DATABASE database
+  [NODE node]
+    TYPE database_type_spec
+    [VERSION “version”]
+    [PHYSICAL “filespec”]
+    [INDEXPREFIX “filespec”]
+    [OPTIONS “options”]
+    [USERCLASS database_access_definition [, database_access_definition ...]]
+</code>
-The syntax for the grid rule is:
+Here is an example of a DATABASE statement on an Omnidex Grid.
 <code>
-  RULE dbname_GRID
+database       "list"
-  {
+  node NODE01
-    MAX_CONCURRENT_NODES 	n
+    type         flatfile
-    UNIQUE_TO_NODE 		“table.column, table.column, …”
+    indexprefix  "{idx/node01/LIST_}"
-    NODE node_name 		[HOST name] [PORT n] [ENVIRONMENT filename]
+  node NODE02
-      DATABASE database_name
+    type         flatfile
-          TABLE table_name PARTITION BY “partition_clause”
+    indexprefix  "{idx/node02/LIST_}"
-  }
+  node NODE03
+    type         flatfile
+    indexprefix  "{idx/node03/LIST_}"
+  node NODE04
+    type         flatfile
+    indexprefix  "{idx/node04/LIST_}"
 </code>
-An example of a grid rule is:
+=== The TABLE statement ===
+The TABLE statement is expanded for each partitioned table to show the table characteristics for each node. The NODE declarations in the TABLE statement are show below.  Consult the documentation on the [[admin:environments:create:oacomp:syntax#table_statement | TABLE statement]] for the complete syntax.
 <code>
+[TABLE table
+  [NODE node]
+    [TYPE table_type_spec]
+    [PHYSICAL "filespec"]
+    [OPTIONS "options"]
+    [CARDINALITY n]
+    [INDEXMAINTENANCE <AUTOMATIC | MANUAL | NONE>]
+    [DATA CACHING <NONE | DYNAMIC>]
+    [PARTITIONED BY "sql_predicates"]
-  rule LIST_GRID
+</code>
-  {
-    max_concurrent_nodes = 4
-    unique_to_node "HOUSEHOLDS.HOUSEHOLD"
-    unique_to_node "HOUSEHOLDS.STATE"
-    unique_to_node "HOUSEHOLDS.ZIP"
-    unique_to_node "INDIVIDUALS.HOUSEHOLD"
-    unique_to_node "INDIVIDUALS.INDIVIDUAL"
-    unique_to_node "INDIVIDUALS.SSN"
-    node node01
+Here is an example of TABLE statements on an Omnidex Grid.
-      database LIST
-        table HOUSEHOLDS
-          partition by "STATE in ('MA','ME','NH','NY','PR','RI','VT',
-                                  'NY','DE','PA','DC','MD','VA','WV') and
-                        ZIP between '00000' and '26999'"
-        table INDIVIDUALS
-          partition by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
-    node node02
+<code>
-      database LIST
+table            "HOUSEHOLDS"
-        table HOUSEHOLDS
+  node NODE01
-          partition by "STATE in ('NC','SC','GA','FL','AL','MS',
+    physical     "{dat/node01/pro*.dat}"
-                                  'TN','KY','OH') and
+    partition by "STATE in ('MA','ME','NH','NY','PR','RI','VT',
-                        ZIP between '27000' and '45999'"
+                            'NY','DE','PA','DC','MD','VA','WV') and
-        table INDIVIDUALS
+                  ZIP between '00000' and '26999'"
-          partition by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
-    node node03
+  node NODE02
-      database LIST
+    physical     "{dat/node02/pro*.dat}"
-        table HOUSEHOLDS
+    partition by "STATE in ('NC','SC','GA','FL','AL','MS',
-          partition by "STATE in ('IN','MI','IA','MN','MT','ND','SD',
+                            'TN','KY','OH') and
-                                  'WI','IL','KS','MO','NE','AR','LA',
+                  ZIP between '27000' and '45999'"
-                                  'OK','TX') and
-                        ZIP between '46000' and '79999'"
+  node NODE03
-        table INDIVIDUALS
+    physical     "{dat/node03/pro*.dat}"
-          partition by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
+    partition by "STATE in ('IN','MI','IA','MN','MT','ND','SD',
+                            'WI','IL','KS','MO','NE','AR','LA',
-    node node04
+                            'OK','TX') and
-      database LIST
+                  ZIP between '46000' and '79999'"
-        table HOUSEHOLDS
-          partition by "STATE in ('AZ','CO','ID','NM','NV','UT','WY',
+  node NODE04
-                                  'CA','HI','AK','OR','WA') and
+    physical     "{dat/node04/pro*.dat}"
-                        ZIP between '80000' and '99999'"
+    partition by "STATE in ('AZ','CO','ID','NM','NV','UT','WY',
-        table INDIVIDUALS
+                            'CA','HI','AK','OR','WA') and
-          partition by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
+                  ZIP between '80000' and '99999'"
-  }
 </code>
+As discussed in the [[admin:grid:partitions#how_are_referential_constraints_handled_such_as_primary_and_foreign_keys| partition scheme]], it is necessary to partition tables so that referential integrity is maintained.  Typically, the parent is partitioned using criteria, as shown for HOUSEHOLDS above, and the child is partitioned along the foreign key.  The PARTITION BY clause for a child shows the join criteria, as shown below:
+<code>
+table            "INDIVIDUALS"
+  node NODE01
+    physical     "{dat/node01/pro*.dat}"
+    partition by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
+  node NODE02
+    physical     "{dat/node02/pro*.dat}"
+    partition by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
+  node NODE03
+    physical     "{dat/node03/pro*.dat}"
+    partition by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
+  node NODE04
+    physical     "{dat/node04/pro*.dat}"
+    partition by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
+</code>
+==== 5. Install and build Omnidex indexes on each node. ====
+It is recommended, though not required, that each node be indexed the same.  This reduces the likelihood of queries taking too long because of a lack of indexing on one node.
+Indexes can be built in parallel.  For maximum performance, it is recommended that the temporary directory, indicated by the TMPDIR environment variable, be different between builds whenever possible.
 ==== 6. Start Omnidex Network Services on each server. ====
@@ Line 132: / Line 185: @@
 Omnidex network services must be started on every machine that contains grid nodes.  No special options are required.
-See how to [[admin:network:overview | configure and start the Omnidex Network Services (OdxNet)]].
+See how to configure and start the [[admin:network:overview | Omnidex Network Services]].

Back to top

admin/grid/creation/oacomp.1258132016.txt.gz · Last modified: 2012/10/26 14:26 (external edit)

Sitemap Recent Changes