Design | Prototype | Development | Testing | Staging | Production | Optimization
Most Omnidex applications involve building indexes on large databases. Some of these applications also involve regular updates or replacements of the data. This data may consist of transaction logs that are produced internally, aggregated monthly or quarterly sales data, or raw data files from list providers or credit agencies. In all of these scenarios, administrators usually assemble and index the data in a staging area, and then deploy it into production.
There are several common strategies for staging and deploying an Omnidex application:
Smaller databases can be staged and deployed on the production server. Smaller databases can be built in a matter of minutes, and this can be scheduled for a time where system load is light. Once the data is assembled and the indexes are built, the data can be copied into the production directory. Alternatively, the application can dynamically switch its database connection strings and simply point to the new data.
Some administrators prefer to have a staging server. A staging server is a separate server that is used to assemble and index the data. Assembling and indexing the data can consume a great deal of system resources and it is beneficial to keep this load away from the production servers. Administrators assemble and index the new data, test to insure the new data is ready for deployment, and then copy the data and index files to the production server.
For large applications, such as Omnidex Grids, staging servers should be high-performance servers with multiple cores, plenty of memory and extra diskspace. For smaller applications, staging servers can be simpler machines, with just a few cores, a standard amount of memory, and sufficient diskspace. The choice of server should also be based on how quickly the new data must be assembled, indexed, and deployed.
Some administrators maintain two production servers, alternating which server is currently in use. While that server processes the production queries, the other server acts as a staging server. Once the new data is assembled and indexed on the staging server, the roles reverse and the staging server becomes the new production server. Meanwhile, the old production server begins the process of assembling and indexing the next batch of data.
This strategy allows faster deployment since the data and indexes do not need to be copied between servers. This strategy is commonly used for Omnidex Grids as well, since Omnidex Grids benefit from a lot of parallel processing.
Some Omnidex Grids are designed to maintain the last several months of data. As months begin, new nodes are created and old nodes are removed. This strategy is particularly efficient since each month requires only a build of the new data, rather than a rebuild of the entire database.
For especially large databases with a higher frequency of updates, daily nodes can be rolled up into weekly nodes, and weekly nodes can be rolled up into monthly nodes. For example, a database containing the last twelve months of data can consist of 12 nodes for months, 3 or 4 nodes for weeks, and 7 nodes for each day. The daily builds are quite fast, and each week the daily nodes can be rolled up into a weekly node, and the weekly nodes can be rolled up into a monthly node. These rollups can happen at a convenient time, such as a weekend or an evening.
See also: