Administration: Omnidex Features

Omnidex Grids

Creating an Omnidex Grid

Omnidex Grids are straightforward to create. Each grid node is a separate Omnidex environment, with an environment file, indexes and data. For convenience, there are methods to share a single Omnidex environment file across multiple nodes. There are also methods to share data files across multiple nodes.

Before creating an Omnidex Grid, you must first have a partitioning scheme and a distribution plan. These will be needed during the steps of creating an Omnidex Grid.

Omnidex Grids are straightforward to administer. Each grid node is a separate Omnidex environment, with an environment file, indexes and data. For convenience, there are methods to share a single Omnidex environment file across multiple nodes. There are also methods to share data files across multiple nodes.

Steps to create an Omnidex Grid using SQL Statements

1. Create an environment pointing to the original database

Omnidex Grids needs an environment that points to the original database so that the data can be partitioned. This environment is a standard environment that accesses the underlying database.

2. Partition the database

If the database will be logically partitioned, then create the relational views that correlate with the partitioning strategy. If the database will be physically partitioned using an Omnidex Snapshot, then partition the database using the PARTITION statement.

The PARTITION statement is available in the Omnidex SQL language, and is callable from the OdxSQL application. Since the statement is part of the Omnidex SQL language, it may also be called directly from ODBC and JDBC.

The PARTITION statement allows the table to be partitioned one of three ways, 1) into a specific number of equally sized partitions, 2) into separate partitions based on a partition qualifier consisting of a column or a SQL expression, or 3) a combination of these two steps. In the third case, Omnidex will evaluate the cardinality of the partition qualifier and group values together to comprise the specific number of partitions in as equal size as possible. The syntax of the PARTITION statement is:

PARTITION table [INTO n [PARTITIONS]] [BY column] 
                [IN path] 
                [WITH options]

An example of a statement to partition a table is:

PARTITION LIST BY “STATE || SUBSTRING(ZIP from 1 for 2)” 
  IN D:\grid\data

3. Distribute the partitioned data across the grid.

If the database will be logically partitioned, then make sure each grid server has access to the database. If the database will be physically partitioned, then distribute both the partitioned and unpartitioned data to the grid nodes. If each node will be in its own directory, then the data files can be copied or symbolically linked to the correct locations. It is also possible for multiple nodes to share the same physical directories as long as the partitioned data can be differentiated.

4. Create the environment file for the Omnidex Grid.

The environment file for the Omnidex Grid is a normal environment file; however, it has additional information that describes the locations of the nodes and the partitioning scheme. There are changes that need to be made for the ENVIRONMENT statement, the DATABASE statement and the TABLE statements. Note that older versions of Omnidex Grids had a RULE statement at the bottom, and that is no longer necessary.

The CREATE ENVIRONMENT statement

The CREATE ENVIRONMENT statement is used to declare the grid nodes and their connection information. The NODE declarations in the CREATE ENVIRONMENT statement are show below. Consult the documentation on the CREATE ENVIRONMENT statement for the complete syntax.

CREATE ENVIRONMENT environment
  [MAX_THREADS n]
  [NODE node
    [<PARTITIONED | UNPARTITIONED>]
    [HOST <LOCALHOST | host> [PORT n]]
    [ENVIRONMENT “filename”]]

Here is an example of a CREATE ENVIRONMENT statement for an Omnidex Grid.

create environment "list_env"
  maxthreads  4
  node NODE01 partitioned
  node NODE02 partitioned
  node NODE03 partitioned 
  node NODE04 partitioned
  ...
The CREATE DATABASE statement

The CREATE DATABASE statement is used to declare the database characteristics for each node. The NODE declarations in the CREATE DATABASE statement are show below. Consult the documentation on the CREATE DATABASE statement for the complete syntax.

CREATE DATABASE database
  [NODE node]
    TYPE type
    [SUBTYPE subtype]
    [PHYSICAL "physical"]
    [USER “user”]
    [PASSWORD “password”]
    [INDEX_DIRECTORY “directory”]
    [INDEX_MAINTENANCE index_maintenance]
    ...

Here is an example of a CREATE DATABASE statement for an Omnidex Grid.

create database       "list"
  node NODE01
    type              flatfile
    index_directory   "idx/node01/LIST_"
  node NODE02
    type              flatfile
    index_directory   "idx/node02/LIST_"
  node NODE03
    type              flatfile
    index_directory   "idx/node03/LIST_"
  node NODE04
    type              flatfile
    index_directory   "idx/node04/LIST_"
The CREATE TABLE statement

The CREATE TABLE statement is used to declare the table characteristics for each node for each partitioned table in the grid. The NODE declarations in the CREATE TABLE statement are show below. Consult the documentation on the CREATE TABLE statement for the complete syntax.

CREATE TABLE table
  [NODE node]
    [OPTIONS “options”]
    [PHYSICAL “physical”]
    [AUTOFILTER “criteria”]
    [DATA_CACHING <DYNAMIC | NONE>]
    [PARTITION_BY “criteria”]
    [INDEX_MAINTENANCE index_maintenance]
    ...

Here is an example of a CREATE TABLE statements for an Omnidex Grid.

create table     "HOUSEHOLDS"
  node NODE01
    physical     "dat/node01/pro*.dat"
    partition_by "STATE in ('MA','ME','NH','NY','PR','RI','VT',
                            'NY','DE','PA','DC','MD','VA','WV') and 
                  ZIP between '00000' and '26999'"

  node NODE02
    physical     "dat/node02/pro*.dat"
    partition_by "STATE in ('NC','SC','GA','FL','AL','MS',
                            'TN','KY','OH') and 
                  ZIP between '27000' and '45999'"
 
  node NODE03
    physical     "dat/node03/pro*.dat"
    partition_by "STATE in ('IN','MI','IA','MN','MT','ND','SD',
                            'WI','IL','KS','MO','NE','AR','LA',
                            'OK','TX') and 
                  ZIP between '46000' and '79999'"

  node NODE04
    physical     "dat/node04/pro*.dat"
    partition_by "STATE in ('AZ','CO','ID','NM','NV','UT','WY',
                            'CA','HI','AK','OR','WA') and 
                  ZIP between '80000' and '99999'"

As discussed in the partition scheme, it is necessary to partition tables so that referential integrity is maintained. Typically, the parent is partitioned using criteria, as shown for HOUSEHOLDS above, and the child is partitioned along the foreign key. The PARTITION BY clause for a child shows the join criteria, as shown below:

create table     "INDIVIDUALS"
  node NODE01
    physical     "dat/node01/pro*.dat"
    partition_by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"

  node NODE02
    physical     "dat/node02/pro*.dat"
    partition_by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"
 
  node NODE03
    physical     "dat/node03/pro*.dat"
    partition_by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"

  node NODE04
    physical     "dat/node04/pro*.dat"
    partition_by "HOUSEHOLDS.HOUSEHOLD = INDIVIDUALS.HOUSEHOLD"

5. Install and build Omnidex indexes on each node.

It is recommended, though not required, that each node be indexed the same. This reduces the likelihood of queries taking too long because of a lack of indexing on one node.

Indexes can be built in parallel. For maximum performance, it is recommended that the temporary directory, indicated by the TMPDIR environment variable, be different between builds whenever possible.

6. Start Omnidex Network Services on each server.

Omnidex network services must be started on every machine that contains grid nodes. No special options are required.

See how to configure and start the Omnidex Network Services.

7. Perform queries against the controller.

Applications should connect to the grid controller as though it was a standard Omnidex environment.

Additional Resources

 
Back to top
admin/features/grids/creation.txt · Last modified: 2016/06/28 22:38 (external edit)