Integration: Raw Data Files

Overview

Omnidex is frequently used to allow high-performance searches against raw data files. Sometimes, the raw data files come from an Omnidex Snapshot, which is an extracted copy of data from an underlying relational database. Other times, the raw data files are received from vendors such as credit services and list providers. Other times, the data is the output of an internal application, such as a log file or transaction file. Regardless of the source of the data, Omnidex will allow raw data files to be treated as a table within a database.

Omnidex: A NoSQL Database and a Big Data Solution

Omnidex and raw data files are a well-tested and mature implementation of a NoSQL database. The NoSQL movement is a loose term describing databases that search data outside of the traditional relational database. NoSQL databases are designed to improve performance by not adhering to the ACID requirements for databases governing Atomicity, Consistency, Isolation and Durability. Raw data files are a classic implementation of this principle.

Omnidex is also particulary strong as a Big Data solution. Raw data files can be extraordinarily large, yet can be easily accessed using standard SQL statements through basic ODBC and JDBC connections. Omnidex can even create a logical table out of a large collection of independent data files, allowing data to come and go without the costly overhead of importing and deleting from relational tables.

Omnidex and Raw Data Files

Using Omnidex on raw data files has many advantages:

  • Access the rest of your data - Most businesses miss out on the data that is stored outside their relational database. They simply don't have a way to search it. Omnidex turns raw data files into instant databases.
  • No need for data imports - Omnidex directly searches raw data files, flat files, and delimited files without having to import them into a relational database. Ease the load on your servers and search the raw data files directly.
  • Deploy your data while it is fresh - Pushing raw data files through a relational database means you have to wait to deploy it. Index it directly and make it available while it is still fresh.
  • Better Performance - Omnidex usually performs better on raw data files than on relational databases. In general, Omnidex scans raw data files about three times faster than relational tables, given their simple structure and lower overhead.
  • Built-in portability - A database consisting of raw data files and Omnidex indexes can be easily copied to other servers. Simply copy a directory of files to wherever you want.
  • Save license fees - Save some money. The license fees for relational databases are not required on servers that simply host raw data files and Omnidex indexes.

Omnidex can index and search raw data files using SQL statements, available through standard interfaces like ODBC and JDBC. Applications usually do not know that they are even accessing raw data files, except for an increase in performance. For many applications, the possible advantages of a relational databases are overwhelmed by the benefits of raw data files.

Additional Resources

See also:

 
Back to top
integration/rawdata/home.txt ยท Last modified: 2016/06/28 22:38 (external edit)