Integration: Raw Data Files

Delimited Files

Limitations of Delimited Files

Delimited files are often the most portable and flexible data file to be used, but they do have some limitations. These limitations are described below:

  • Datatypes - Delimited files can store nearly every datatype supported by Omnidex, including proprietary datatypes such as ORACLE DATETIME or ODBC DATETIME. An exception to this is the relational datatypes, VARCHAR and CLOB. These datatypes cannot be represented in delimited files because they require separate storage for length and NULL indicators. For delimited files, the STRING and CHARACTER datatypes should be used instead.
  • NULLs - Delimited files are not capable of differentiating a NULL column as is done in relational databases. Omnidex will treat empty fields as NULL fields to compensate for this. For character-class datatypes, an IS NULL test will qualify any row containing an empty space in that column. For binary-class datatypes, an IS NULL test will qualify any row containing a zero in that column.
  • Deletes and Updates - Omnidex supports INSERT statements against delimited files, but it cannot support the UPDATE or DELETE statement. If UPDATE or DELETE statements are required, consider using Omnidex Standalone Tables.
  • Data Conversions - Delimited files store all data in character format, including binary data such as integers, floating point data, and dates. These conversions add overhead to the parsing of the data. Files with a large number of binary fields can achieve better performance using fixed-length files or Omnidex Standalone Tables
  • Security - Delimited files are not encrypted as they might be in a relational database. Care should be taken with file system security to properly protect the data.

Despite these limitations, delimited files provide a fast, flexible, and portable solution that brings performance and ease to an application.

Additional Resources

See also:

Back to top
integration/rawdata/delimited/limitations.txt ยท Last modified: 2016/06/28 22:38 (external edit)