Integration: Raw Data Files

Overview | File Types | Environments | Queries | Updates | Generating Data Files | Integration Guides


Overview

Omnidex is frequently used to allow high-performance searches against raw data files. Sometimes, the raw data files come from an Omnidex Snapshot, which is an extracted copy of data from an underlying relational database. Other times, the raw data files are received from vendors such as credit services and list providers. Other times, the data is the output of an internal application, such as a log file or transaction file. Regardless of the source of the data, Omnidex will allow raw data files to be treated as a table within a database.

Omnidex: A NoSQL Database and a Big Data Solution

Omnidex and raw data files are a well-tested and mature implementation of a NoSQL database. The NoSQL movement is a loose term describing databases that search data outside of the traditional relational database. NoSQL databases are designed to improve performance by not adhering to the ACID requirements for databases governing Atomicity, Consistency, Isolation and Durability. Raw data files are a classic implementation of this principle.

Omnidex is also particulary strong as a Big Data solution. Raw data files can be extraordinarily large, yet can be easily accessed using standard SQL statements through basic ODBC and JDBC connections. Omnidex can even create a logical table out of a large collection of independent data files, allowing data to come and go without the costly overhead of importing and deleting from relational tables.

Omnidex and Raw Data Files

Using Omnidex on raw data files has many advantages:

Omnidex can index and search raw data files using SQL statements, available through standard interfaces like ODBC and JDBC. Applications usually do not know that they are even accessing raw data files, except for an increase in performance. For many applications, the possible advantages of a relational databases are overwhelmed by the benefits of raw data files.

Next

Additional Resources

See also: