This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
admin:indexing:powersearch:custom [2011/01/24 18:27] els |
admin:indexing:powersearch:custom [2012/10/26 14:57] (current) |
||
|---|---|---|---|
| Line 16: | Line 16: | ||
| ---- | ---- | ||
| - | ==== Creating Custom Synonym Lists ==== | + | ==== Custom Synonym Lists ==== |
| - | Synonym lists are an important tool for PowerSearch. Synonym lists allow your search terms to be automatically expanded to make your search more flexibility. The examples on the previous pages used many synonym lists, such as Given Names, Postal Abbreviations, City Names and Area Codes. Omnidex provides a base set of synonym lists as part of the product, but it is also possible to create your own synonym lists. | + | Synonym lists are an important tool for PowerSearch. Synonym lists allow your search terms to be automatically expanded to make your search more flexibile. The examples on the previous pages used many synonym lists, such as Given Names, Postal Abbreviations, City Names and Area Codes. Omnidex provides a base set of synonym lists as part of the product, but it is also possible to create your own synonym lists. |
| - | The synonym lists that are provided with Omnidex are: | + | Synonym lists are usually limited to a specific topic, such as Given Names, or City Abbreviations. It is important to watch for cross-overs between synonyms. For example, it would be prudent to keep city, state and country abbreviations in separate lists. LA as a city code is an abbreviation for Los Angeles; however, as a state code, it is an abbreviation for Louisiana. In these situations, each column should use its own managed synonym list so that overlap does not occur. |
| + | ==== The Synonym List Library ==== | ||
| - | ^LIST^DESCRIPTION^ | + | Before creating your own synonym lists, be sure to check the [[appendix:synonyms|synonym lists]] that are provided with Omnidex. The list you need may have already been created, or there may be a similar list that you can use as a starting point. If you do create your own synonym list, consider whether it would benefit the broader Omnidex community. If you would like to submit a synonym list for inclusion in the product, simply send it to [[appendix:contactus|Technical Support]] with a note saying that you are contributing it to the product. |
| - | |**Geography and Addresses**\\ || | + | |
| - | |COUNTRY_CODES|ISO Standard country codes| | + | |
| - | |STATE_CODES|USPS state codes| | + | |
| - | |CANADIAN_PROVINCES|Canadian province codes| | + | |
| - | |CITY_ABBR|Common abbreviations used in city names| | + | |
| - | |STREET_SUFFIXES|USPS standard abbreviations used in street addresses| | + | |
| - | |SECONDARY_UNIT_ABBR|USPS standard abbreviations used for apartments, suites, etc.| | + | |
| - | |DIRECTIONS|Common abbreviations used for directions of the compass.| | + | |
| - | |NUMERALS|Correlation of numerical and textual numbers, such as 10 and ten.| | + | |
| - | |ORDINALS|Correlation of numerical and textual ordinals, such as 1st and first.| | + | |
| - | |SALUTATION_ABBR|Standard abbreviations used in salutations| | + | |
| - | |MILTARY_RANK_ABBR|Standard abbreviations used for military ranks| | + | |
| - | |ORGANIZATION_ABBR|Standard abbreviations used in the names of organizations| | + | |
| - | |AIRPORT_CODES_US|Standard airport codes within the United States| | + | |
| - | |AIRPORT_CODES_INTL|Standard airport codes outside of the United States| | + | |
| - | |ALL_INDIVIDUAL_ADDRESSEES*|Synonyms appropriate for addressees that are individual people, as opposed to organizations. This list is a composite of SALUTATION_ABBR and MILITARY_RANK_ABBR.| | + | |
| - | |ALL_ADDRESSEES*|Synonyms appropriate for addressees, including people and organizations. This list is a composite of SALUTATIONS_ABBR, MILITARY_RANK_ABBR and ORGANIZATION_ABBR.| | + | |
| - | |ALL_ADDRESS_LINES*|Synonyms appropriate for address lines containing street address, but not city, state, zip and country. This list is a composite of STREET_SUFFIXES, SECONDARY_UNIT_ABBR, DIRECTIONS, NUMERALS and ORDINALS| | + | |
| - | |ALL_FREEFORM_ADDRESSES*|Synonyms appropriate for address lines containing street address, city, state and zip, but not country. This list is a composite of STATE_CODES, CANADIAN_PROVINCES, CITY_ABBR, STREET_SUFFIXES, SECONDARY_UNIT_ABBR, DIRECTIONS, NUMERALS and ORDINALS.| | + | |
| - | |ALL_AIRPORTS*|All standard airport codes, both within and outside the United States. This list is a composite of AIRPORT_CODES_US and AIRPORT_CODES_INTL.| | + | |
| - | |ALL_GEOGRAPHY_ADDRESSES*|All geographical and address synonyms. This list is a composite of all lists in this section.| | + | |
| - | |**Proper Names**\\ || | + | |
| - | |FEMALE_GIVEN_NAMES|Variations for female-only given names.| | + | |
| - | |MALE_GIVEN_NAMES|Variations for male-only given names.| | + | |
| - | |EXPANDED_FEMALE_GIVEN_NAMES|Expanded variations for male or female given names. This list is a superset of FEMALE_GIVEN_NAMES, and also includes more unusual variations, as well as correlated nicknames sharing the same base name (eg. Elizabeth = Liz and Beth, therefore Liz = Beth).| | + | |
| - | |EXPANDED_MALE_GIVEN_NAMES|Expanded variations for male or female given names. This list is a superset of MALE_GIVEN_NAMES, and also includes more unusual variations, as well as correlated nicknames sharing the same base name (eg. William = Bill and Will, therefore Bill = Will).| | + | |
| - | |SURNAMES|Variations for surnames. Generally combined with use of phonetic functions.| | + | |
| - | |ALL_GIVEN_NAMES*|All variations of given names. This list is a composite of FEMALE_GIVEN_NAMES and MALE_GIVEN_NAMES.| | + | |
| - | |EXPANDED_GIVEN_NAMES*|All variations of expanded given names. This list is a composite of EXPANDED_FEMALE_GIVEN_NAMES and EXPANDED_MALE_GIVEN_NAMES.| | + | |
| - | |ALL_PROPER_NAMES*|All variations of proper names. This list is a composite of all lists in this section.| | + | |
| - | |**Acronyms**\\ || | + | |
| - | |GENERAL_ACRONYMS|General acronyms found in general use.| | + | |
| - | |ORGANIZATION_ACRONYMS|Acronyms for companies and organizations.| | + | |
| - | |GOV_ABBR_GPO|Government acronyms from the Government Printing Office.| | + | |
| - | |GOV_ABBR_IUPUI|Government acronyms from Indiana University/Perdue University.| | + | |
| - | |EMAIL_CHAT_ACRONYMS|Acronyms commonly used in emails and chat rooms.| | + | |
| - | |ALL_ACRONYMS*|All acronyms. This list is a composite of all lists in this section.| | + | |
| - | |**Abbreviations**\\ || | + | |
| - | |MEASURE_BASIC_ABBR|Abbreviations for basic measures| | + | |
| - | |MEASURE_LENGTH_ABBR|Abbreviations for measurements of lengths.| | + | |
| - | |MEASURE_AREA_ABBR|Abbreviations for measurements of area.| | + | |
| - | |MEASURE_LIQUID_VOLUME_ABBR|Abbreviations for measurements of liquid volume.| | + | |
| - | |MEASURE_DRY_VOLUME_ABBR|Abbreviations for measurements of dry volume.| | + | |
| - | |MEASURE_WEIGHT_ABBR|Abbreviations for measurements of weight.| | + | |
| - | |MEASURE_ENERGY_ABBR|Abbreviations for measurements of energy.| | + | |
| - | |MEASURE_TIME_ABBR|Abbreviations for measurements of time. Note that this list does not include the standard abbreviations for the days of the week and the months. Those abbreviations are found in their respective lists.| | + | |
| - | |TIME_PERIODS|Mnemonics for common time periods, correlated to their appropriate date range. | | + | |
| - | |MONTH_ABBR|Standard abbreviations for the months of the year.| | + | |
| - | |DAYS_OF_WEEK_ABBR|Standard abbreviations for the days of the week.| | + | |
| - | |TIME_ZONE_ABBR|Standard abbreviations for the worldwide time zones.| | + | |
| - | |ALL_MEASURES*|All abbreviations of measurements. This list is a composite of MEASURE_BASIC_ABBR, MEASURE_LENGTH_ABBR, MEASURE_AREA_ABBR, MEASURE_LIQUID_VOLUME_ABBR, MEASURE_DRY_VOLUME_ABBR, MEASURE_WEIGHT_ABBR, MEASURE_ENERGY_ABBR and MEASURE_TIME_ABBR.| | + | |
| - | |ALL_CALENDAR*|All calendar-related abbreviations. This list is a composite of MONTH_ABBR and DAYS_OF_WEEK_ABBR.| | + | |
| - | |ALL_TIME*|All time-related abbreviations. This list is a composite of MEASURE_TIME_ABBR, MONTH_ABBR, DAYS_OF_WEEK_ABBR and TIME_ZONE_ABBR.| | + | |
| - | |ALL_ABBREVIATIONS*|All abbreviations. This list is a composite of all lists in this section.| | + | |
| - | |**Medical**\\ || | + | |
| - | |MEDICAL_ABBR|Common medical abbreviations.| | + | |
| - | |DRUG_BRANDS_DISCN|Correlation of discontinued drug brands and their ingredients from the FDA's Orange Book.| | + | |
| - | |DRUG_BRANDS_OTC|Correlation of over-the-counter drug brands and their ingredients from the FDA's Orange Book.| | + | |
| - | |DRUG_BRANDS_RX|Correlation of prescription drug brands and their ingredients from the FDA's Orange Book.| | + | |
| - | |DRUG_INGR_DISCN|Correlation of the ingredients of discontinued drugs and their brands from the FDA's Orange Book.| | + | |
| - | |DRUG_INGR_OTC|Correlation of the ingredients of over-the-counter drugs and their brands from the FDA's Orange Book.| | + | |
| - | |DRUG_INGR_RX|Correlation of the ingredients of prescription drugs and their brands from the FDA's Orange Book.| | + | |
| - | |COMMON_DRUGS|Correlation of common drug names and their generic equivalents.| | + | |
| - | |ALL_DRUG_BRANDS*|All drugs from FDA Orange Book, by brand. This list is a composite of DRUG_BRANDS_DISCN, DRUG_BRANDS_OTC and DRUG_BRANDS_RX.| | + | |
| - | |ALL_DRUG_INGR*|All drugs from FDA Orange Book, by ingredien. This list is a composite of DRUG_INGR_DISCN, DRUG_INGR_OTC and DRUG_INGR_RX.| | + | |
| - | |FDA_ORANGE_BOOK*|All drugs from FDA Orange Book, by brand and ingredient. This list is a composite of DRUG_BRANDS_DISCN, DRUG_BRANDS_OTC, DRUG_BRANDS_RX, DRUG_INGR_DISCN, DRUG_INGR_OTC and DRUG_INGR_RX.| | + | |
| - | |ALL_DRUGS*|All drugs from FDA Orange Book plus other lists of drugs. This list is a composite of DRUG_BRANDS_DISCN, DRUG_BRANDS_OTC, DRUG_BRANDS_RX, DRUG_INGR_DISCN, DRUG_INGR_OTC, DRUG_INGR_RX and COMMON_DRUGS.| | + | |
| - | |ALL_MEDICAL*|All medical synonyms. This list is a composite of all lists in this section.| | + | |
| - | |**Science**\\ || | + | |
| - | |CERN_ABBR|Acronyms and abbreviations appropriate for the CERN environment.| | + | |
| - | |BASIC_ELEMENTS|Correlation of abbreviations and elements from the Periodic Table of Elements | | + | |
| - | |COMMON_CHEMICAL_COMPOUNDS|Correlation of composition and names for common chemical compounds.| | + | |
| - | |ALL_SCIENCE*|All science synonyms. This list is a composite of all lists in this section.| | + | |
| - | |**General**\\ || | + | |
| - | |ALL_SYNONYMS*|This list is a composite of all lists in the sections above.| | + | |
| + | ==== The Synonym List File Layout ==== | ||
| + | |||
| + | Creating a new synonym list is straightforward. It is simply a tab-delimited file with four columns. It can be created using a text editor such as 'Notepad' on Windows or 'vi' on UNIX. It can also be created in a spreadsheet program such as Microsoft Excel, which allows a file to be saved as a Tab-Delimited File. | ||
| + | |||
| + | The record layout of the synonym table consists of four columns: | ||
| + | |||
| + | ^ Column Name ^ Datatype ^ | ||
| + | | $LIST | CHARACTER(32) | | ||
| + | | $WORD | STRING(127) | | ||
| + | | $REPLACEMENT | STRING(4094) | | ||
| + | |$COMMENTS | STRING(255) | | ||
| + | |||
| + | |||
| + | === $LIST === | ||
| + | |||
| + | The name of the list, repeated for each row in the list. This name will be referenced in the $CONTAINS clause using the syntax, 'SYNONYMS=list'. Be sure to choose a list that does not conflict with an existing list in the library. | ||
| + | |||
| + | === $WORD === | ||
| + | |||
| + | The word or phrase for which synonyms are being created. Phrases should be enclosed in double quotes. | ||
| + | |||
| + | === $REPLACEMENT === | ||
| + | |||
| + | The replacement string to be used as a synonym list. Normally, this is a comma-delimited list of synonyms; however, it can be any of the following choices, intermingled and in any order. | ||
| + | |||
| + | Replacements may consist of any of the following choices. Replacements may also be intermingled, and occur in any order. | ||
| + | |||
| + | == Words and Phrases == | ||
| + | |||
| + | Replacements can be a word or phrase to be used in place of the current word. Note that for the synonyms to include the current word itself, it must be included in the replacement text. Words may be separated by commas. Phrases must be enclosed in double-quotation marks. For example: | ||
| + | |||
| + | ^ $LIST ^ $WORD ^ $REPLACEMENT ^ $COMMENTS ^ | ||
| + | | CITIES | LA | LA, "Los Angeles" | | | ||
| + | | CITIES | SF | SF, "San Francisco", "Santa Fe" | | | ||
| + | | STATES | LA | LA, Louisiana | | | ||
| + | | STATES | NM | NM, "New Mexico" | | | ||
| + | |||
| + | |||
| + | == Qualification Criteria == | ||
| + | |||
| + | Replacements can be qualification criteria, indicated by enclosing the entire replacement string in parentheses. These criteria may include Boolean operators and nested parentheses. For example: | ||
| + | |||
| + | ^ $LIST ^ $WORD ^ $REPLACEMENT ^ $COMMENTS ^ | ||
| + | | DATE_RANGES | FISCAL_2009 | (Between 7/1/2008 and 6/30/2009) | | | ||
| + | | DATE_RANGES | FISCAL_2010 | (Between 7/1/2009 and 6/30/2010) | | | ||
| + | | DATE_RANGES | FISCAL_2011 | (Between 7/1/2010 and 6/30/2011) | |== | ||
| + | |||
| + | == Pointers to Other Entries == | ||
| + | |||
| + | Replacements can be pointers to other entries within the same list. Pointers are indicated by prefixing the word with a greater-than sign (>). Pointers are allowed to be nested. For example: | ||
| + | |||
| + | ^ $LIST ^ $WORD ^ $REPLACEMENT ^ $COMMENTS ^ | ||
| + | | FIRST_NAMES | FRED | >Fredrick | | | ||
| + | | FIRST_NAMES | FREDRICK | Fredrick, Fred, Rick | | | ||
| + | | FIRST_NAMES | RICK | Rick, Ricky, Richard, Dick, >Fredrick | | | ||
| + | |||
| + | |||
| + | ==== Installing a New Synonym List ==== | ||
| + | |||
| + | Installing a new synonym list is as simple as saving the file in the synonym directory. The synonym directory is as follows: | ||
| + | |||
| + | Windows: | ||
| + | |||
| + | %OMNIDEX_HOME%\config\english\synonyms | ||
| + | |||
| + | UNIX: | ||
| + | |||
| + | $OMNIDEX_HOME/config/english/synonyms | ||
| + | |||
| + | After the file has been installed, it must be indexed. This is done using the following command: | ||
| + | |||
| + | Windows: | ||
| + | |||
| + | cd %OMNIDEX_HOME%\config | ||
| + | build_config.bat | ||
| + | |||
| + | UNIX: | ||
| + | |||
| + | cd $OMNIDEX_HOME/config | ||
| + | build_config.ksh | ||
| + | |||
| + | ==== Testing a New Synonym List ==== | ||
| + | |||
| + | Once a synonym list has been installed, it can be tested using the following command in OdxSQL. In this example, replace the values in angle brackets with values appropriate to your database. | ||
| + | |||
| + | connect <environment> | ||
| + | lookup $contains(<table>.<column>, 'criteria', 'SYNONYMS=<list>') | ||
| + | |||
| + | As an example, this same command is shown running against the ALL_GIVEN_NAMES synonym list: | ||
| + | |||
| + | > connect simple | ||
| + | Connected to D:\class\lab2\simple.xml | ||
| + | > lookup $contains(individuals.name, 'William', 'synonyms=ALL_GIVEN_NAMES') | ||
| + | IN (William, Bill, Billy, Will, Williams, Willie, Willis, Wilson) | ||
| - | $LIST CHARACTER(32) | ||
| - | $WORD C STRING(127) | ||
| - | $REPLACEMENT C STRING(4094) | ||
| - | $COMMENTS C STRING(255) | ||