Differences

This shows you the differences between two versions of the page.

--- admin:features:grids:creation [2011/01/10 22:25]
els
+++ admin:features:grids:creation [2016/06/28 22:38] (current)
@@ Line 1: / Line 1: @@
+~~NOTOC~~
 {{page>:top_add&nofooter&noeditbtn}}
@@ Line 5: / Line 7: @@
 ===== Omnidex Grids =====
-[[admin:features:grids:home|Overview]] -> [[admin:features:grids:partitions|Partitioning Scheme]] -> [[admin:features:grids:distribution|Distribution Plan]] -> **[[admin:features:grids:creation|Grid Creation]]**
+[[admin:features:grids:home|Overview]] | [[admin:features:grids:partitions|Partitioning Scheme]] | [[admin:features:grids:distribution|Distribution Plan]] | **[[admin:features:grids:creation|Grid Creation]]**
 ----
@@ Line 15: / Line 17: @@
 Before creating an Omnidex Grid, you must first have a [[admin:features:grids:partitions | partitioning scheme]] and a [[admin:features:grids:distribution | distribution plan]].  These will be needed during the steps of creating an Omnidex Grid.
-There are several ways to create Omnidex Grids:
+Omnidex Grids are straightforward to administer.  Each grid node is a separate Omnidex environment, with an environment file, indexes and data.  For convenience, there are methods to share a single Omnidex environment file across multiple nodes.  There are also methods to share data files across multiple nodes.
-  * To create an Omnidex Grid in an interactive tool, use the [[admin:features:grids:creation:odxadmin| Omnidex Administrator]].
-  * To create an Omnidex Grid with scripts, use [[admin:features:grids:creation:sql | Omnidex SQL statements]] such as CREATE ENVIRONMENT, CREATE DATABASE and CREATE TABLE.
+==== Steps to create an Omnidex Grid using SQL Statements ====
+=== 1. Create an environment pointing to the original database ===
+Omnidex Grids needs an environment that points to the original database so that the data can be partitioned.  This environment is a standard environment that accesses the underlying database.
+=== 2. Partition the database ===
+If the database will be logically partitioned, then create the relational views that correlate with the partitioning strategy.  If the database will be physically partitioned using an Omnidex Snapshot, then partition the database using the PARTITION statement.
+The PARTITION statement is available in the Omnidex SQL language, and is callable from the OdxSQL application.  Since the statement is part of the Omnidex SQL language, it may also be called directly from ODBC and JDBC.
+The PARTITION statement allows the table to be partitioned one of three ways, 1) into a specific number of equally sized partitions, 2) into separate partitions based on a partition qualifier consisting of a column or a SQL expression, or 3) a combination of these two steps.  In the third case, Omnidex will evaluate the cardinality of the partition qualifier and group values together to comprise the specific number of partitions in as equal size as possible.  The syntax of the PARTITION statement is:
+  PARTITION table [INTO n [PARTITIONS]] [BY column]
+                  [IN path]
+                  [WITH options]
+An example of a statement to partition a table is:
+  PARTITION LIST BY “STATE || SUBSTRING(ZIP from 1 for 2)”
+    IN D:\grid\data
+=== 3.	Distribute the partitioned data across the grid.  ===
+If the database will be logically partitioned, then make sure each grid server has access to the database.  If the database will be physically partitioned, then distribute both the partitioned and unpartitioned data to the grid nodes.  If each node will be in its own directory, then the data files can be copied or symbolically linked to the correct locations.  It is also possible for multiple nodes to share the same physical directories as long as the partitioned data can be differentiated.
+=== 4. Create the environment file for the Omnidex Grid. ===
+The environment file for the Omnidex Grid is a normal environment file; however, it has additional information that describes the locations of the nodes and the partitioning scheme.  There are changes that need to be made for the ENVIRONMENT statement, the DATABASE statement and the TABLE statements.  Note that older versions of Omnidex Grids had a RULE statement at the bottom, and that is no longer necessary.
+== The CREATE ENVIRONMENT statement ==
+The CREATE ENVIRONMENT statement is used to declare the grid nodes and their connection information.  The NODE declarations in the CREATE ENVIRONMENT statement are show below.  Consult the documentation on the [[dev:sql:statements:create_environment:home | CREATE ENVIRONMENT]] statement for the complete syntax.
+<code>
+CREATE ENVIRONMENT environment
+  [MAX_THREADS n]
+  [NODE node
+    [<PARTITIONED | UNPARTITIONED>]
+    [HOST <LOCALHOST | host> [PORT n]]
+    [ENVIRONMENT “filename”]]
+</code>
+Here is an example of a CREATE ENVIRONMENT statement for an Omnidex Grid.
+<code>
+create environment "list_env"
+  maxthreads  4
+  node NODE01 partitioned
+  node NODE02 partitioned
+  node NODE03 partitioned
+  node NODE04 partitioned
+  ...
+</code>
+== The CREATE DATABASE statement ==
+The CREATE DATABASE statement is used to declare the database characteristics for each node. The NODE declarations in the CREATE DATABASE statement are show below.  Consult the documentation on the [[dev:sql:statements:create_database:home | CREATE DATABASE]] statement for the complete syntax.
+<code>
+CREATE DATABASE database
+  [NODE node]
+    TYPE type
+    [SUBTYPE subtype]
+    [PHYSICAL "physical"]
+    [USER “user”]
+    [PASSWORD “password”]
+    [INDEX_DIRECTORY “directory”]
+    [INDEX_MAINTENANCE index_maintenance]
+    ...
+</code>
+Here is an example of a CREATE DATABASE statement for an Omnidex Grid.
+<code>
+create database       "list"
+  node NODE01
+    type              flatfile
+    index_directory   "idx/node01/LIST_"
+  node NODE02
+    type              flatfile
+    index_directory   "idx/node02/LIST_"
+  node NODE03
+    type              flatfile
+    index_directory   "idx/node03/LIST_"
+  node NODE04
+    type              flatfile
+    index_directory   "idx/node04/LIST_"
+</code>
+== The CREATE TABLE statement ==
+The CREATE TABLE statement is used to declare the table characteristics for each node for each partitioned table in the grid. The NODE declarations in the CREATE TABLE statement are show below.  Consult the documentation on the [[dev:sql:statements:create_table:home | CREATE TABLE]] statement for the complete syntax.
+<code>
+CREATE TABLE table
+  [NODE node]
+    [OPTIONS “options”]
+    [PHYSICAL “physical”]
+    [AUTOFILTER “criteria”]
+    [DATA_CACHING <DYNAMIC | NONE>]
+    [PARTITION_BY “criteria”]
+    [INDEX_MAINTENANCE index_maintenance]
+    ...
+</code>
+Here is an example of a CREATE TABLE statements for an Omnidex Grid.
+<code>
+create table     "HOUSEHOLDS"
+  node NODE01
+    physical     "dat/node01/pro*.dat"
+    partition_by "STATE in ('MA','ME','NH','NY','PR','RI','VT',
+                            'NY','DE','PA','DC','MD','VA','WV') and
+                  ZIP between '00000' and '26999'"
+  node NODE02
+    physical     "dat/node02/pro*.dat"
+    partition_by "STATE in ('NC','SC','GA','FL','AL','MS',
+                            'TN','KY','OH') and
+                  ZIP between '27000' and '45999'"
+  node NODE03
+    physical     "dat/node03/pro*.dat"
+    partition_by "STATE in ('IN','MI','IA','MN','MT','ND','SD',
+                            'WI','IL','KS','MO','NE','AR','LA',
+                            'OK','TX') and
+                  ZIP between '46000' and '79999'"
+  node NODE04
+    physical     "dat/node04/pro*.dat"
+    partition_by "STATE in ('AZ','CO','ID','NM','NV','UT','WY',
+                            'CA','HI','AK','OR','WA') and
+                  ZIP between '80000' and '99999'"
+</code>
+As discussed in the [[admin:features:grids:partitions#how_are_referential_constraints_handled_such_as_primary_and_foreign_keys| partition scheme]], it is necessary to partition tables so that referential integrity is maintained.  Typically, the parent is partitioned using criteria, as shown for HOUSEHOLDS above, and the child is partitioned along the foreign key.  The PARTITION BY clause for a child shows the join criteria, as shown below:
+<code>
+create table     "INDIVIDUALS"
+  node NODE01
+    physical     "dat/node01/pro*.dat"
+    partition_by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
+  node NODE02
+    physical     "dat/node02/pro*.dat"
+    partition_by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
+  node NODE03
+    physical     "dat/node03/pro*.dat"
+    partition_by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
+  node NODE04
+    physical     "dat/node04/pro*.dat"
+    partition_by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
+</code>
+=== 5. Install and build Omnidex indexes on each node. ===
+It is recommended, though not required, that each node be indexed the same.  This reduces the likelihood of queries taking too long because of a lack of indexing on one node.
+Indexes can be built in parallel.  For maximum performance, it is recommended that the temporary directory, indicated by the TMPDIR environment variable, be different between builds whenever possible.
+=== 6. Start Omnidex Network Services on each server. ===
+Omnidex network services must be started on every machine that contains grid nodes.  No special options are required.
+See how to configure and start the [[admin:network:overview | Omnidex Network Services]].
+=== 7. Perform queries against the controller. ===
+Applications should connect to the grid controller as though it was a standard Omnidex environment.
 ====  ====

Back to top

admin/features/grids/creation.1294698357.txt.gz · Last modified: 2016/06/28 22:38 (external edit)

Sitemap Recent Changes