The most common way to process a large volume of updates is to use offline updates. Offline updates simply mean that the data is updated offline and then the indexes are regenerated. While the data is being updated, the indexes are not updated, allowing the indexes to become out of sync with the data. During this time, the Omnidex queries should be avoided on the database. Once the database is updated, the Omnidex indexes are regenerated and the database is brought back online.
This approach is extremely fast. Omnidex can regenerate indexes at a rate of approximately 1 billion index entries an hour per table1. This means that a 10 million row table with 25 indexes can be reindexed in 15 minutes, or an equivalent table with 1 million rows can be reindexed in 90 seconds.
For relational databases, tables are generally updated using the underlying database tools. Some databases provide higher-performing batch tools for updating database in bulk, and nearly all database provide some capability for processing INSERT, DELETE and UPDATE statements. Because these updates are processed without having to update indexes, they can be processed at high rates of speed.
For raw data files, old files are simply replaced with new files. If the new file has been staged, then this replacement step only takes seconds. Omnidex also allows a table to be comprised of multiple physical files that share the same structure. This means that a table can be expanded by simply adding an additional file to the table. These approaches are incredibly fast, far surpassing the time required to insert data into a relational database.
1 An "index entry" represents an indexed column in a single row. If the column contains textual data that is indexed with QuickText or FullText indexes, then an "index entry"" represents a single word or number in an indexed column in a single row. Indexing rates are affected by machine performance, database performance, indexing options and concurrency.
See also: