Release 10.1A: OpenEdge Development:
Programming Interfaces

> ## Using word-break tables

> You can create word-break tables that specify word separators using a rich set of criteria. To specify and work with word-break tables involves:

> - Specifying word delimiter attributes.

> - Understanding the syntax of word-break tables.

> - Compiling word-break tables.

> - Associating compiled word-break tables with databases.

> - Rebuilding word indexes.

> - Providing access to compiled word-break tables.

> ##### Specifying word delimiter attributes

> As mentioned previously, to break down the contents of a word-indexed field into individual words, Progress needs to know which characters delimit words and which do not. The distinction can be subtle and sometimes depends on context. For example, consider the function of the dot in the character strings in Table 1–5.

> Table 1–5: Is the dot a word delimiter?
> Character string | Function of the dot | Is the dot a word delimiter?
> --- | --- | ---
> “Balance is $25,125.95” | Decimal separator | No
> “Shipment not received.Call customs broker” | Period at end of sentence. | Yes

> In the first character string, the dot functions as a decimal point and does not divide one word from another. Thus, you can query on the word “$25,125.95.” In the second character string, by contrast, the dot functions as a period, dividing the word “received” from the word “call.”

> To help define word delimiters systematically while allowing for contextual variation, Progress provides eight word delimiter attributes, which you can use in word-break tables. The eight word delimiter attributes appear in Table 1–6.

> Table 1–6: Word delimiter attributes
> Word delimiter attribute | Description | Default
> --- | --- | ---
>     LETTER
> | Always part of a word. | Assigned to all characters that the current attribute table defines as letters. In English, these are the uppercase characters A–Z and the lowercase characters a–z.
>     DIGIT
> | Always part of a word. | Assigned to the characters 0–9.
>     USE_IT
> | Always part of a word. | Assigned to the following characters: Dollar sign ($) Percent sign (%) Number sign (#) At symbol (@) Underline (_)
>     BEFORE_LETTER
> | Part of a word only if followed by a character with the LETTER attribute. Else, treated as a word delimiter. | –
>     BEFORE_DIGIT
> | Treated as part of a word only if followed by a character with the DIGIT attribute. | Assigned to the following characters: Period (.) Comma (,) Hyphen (–) For example, “12.34” is one word, but “ab.cd” is two words.
>     BEFORE_LET_DIG
> | Treated as part of a word only if followed by a character with the `LETTER` or `DIGIT` attribute. | -
>     IGNORE
> | Ignored. | Assigned to the apostrophe (’). For example, “John’s” is equivalent to “Johns.”
>     TERMINATOR
> | Word delimiter. | Assigned to all other characters.

> ##### Understanding the syntax of word-break tables

> Word delimiter attributes form the heart of word break tables, and you specify them using the following syntax:

> Syntax
>     [ #define symbolic-name symbol-value ] ...
>     [ Version = 9
>        Codepage = codepage-name
>        wordrules-name = wordrules-name
>        type = table-type
>     ]
>     word_attr =
>     {
>       { char-literal | hex-value | decimal-value } , word-delimiter-attribute
>           [ , { char-literal | hex-value | decimal-value }
>                , word-delimiter-attribute ] ...
>     };

> ``symbolic-name``

> The name of a symbol.

> For example: `DOLLAR-SIGN`

> ``symbol-value``

> The value of the symbol.

> For example: ’$’

> Note: Although some versions of Progress let you compile word-break tables that omit all items within the second pair of square brackets, Progress Software Corporation (PSC) recommends that you always include these items. If the source-code version of a compiled word-break table lacks these items, and the associated database is not so large as to make this unfeasible, PSC recommends that you add these items to the table, recompile the table, reassociate the table with the database, and rebuild the indexes.

> ``codepage-name``

> The name, not surrounded by quotes, of the code page the word-break table is associated with. The maximum length is 20 characters.

> For example: UTF–8

> ``wordrules-name``

> The name, not surrounded by quotes, of the compiled word-break table. The maximum length is 20 characters.

> For example: utf8sample

> ``table-type``

> The number 2.

> Note: Some versions of Progress allow a table type of 1. Although this is still supported, Progress Software Corporation (PSC) recommends, if feasible, that you change the table type to 2, recompile the word-break table, reassociate it with the database, and rebuild the indexes.

> ``char-literal``

> A character within single quotes or a `symbolic`–`name,` which represents a character in the code page.

> For example: ’#’

> ``hex-literal``

> A hexadecimal value or a `symbolic`–`name,` which represents a character in the code page.

> For example:0xAC

> ``decimal-literal``

> A decimal value or a `symbolic`–`name`, which represents a character in the code page.

> For example: 39

> ``word-delimiter-attribute``

> In what context the character is a word delimiter. You can use one of the following:

> - `LETTER`
> - `DIGIT`
> - `USE_IT`
> - `BEFORE_LETTER`
> - `BEFORE_DIGIT`
> - `BEFORE_LET_DIG`
> - `IGNORE`
> - `TERMINATOR`

> ##### Examples of word-break tables

> The following is an example of a word-break table for Unicode:

>     /* a word-break table for Unicode */
>     #define DOLLAR-SIGN ’$’
>     Version = 9
>     Codepage = utf-8
>     wordrules-name = utf8sample
>     type = 2
>     word_attr =
>     {
>       ’.’,         BEFORE_DIGIT,
>       ’,’,         BEFORE-DIGIT,
>       0x2D,        BEFORE_DIGIT,
>       39,          IGNORE,
>       DOLLAR-SIGN, USE-IT,
>       ’%’,         USE-IT,
>       ’#’,         USE_IT,
>       ’@’,         USE-IT,
>       ’_’,         USE_IT,
>     };

> As the preceding example illustrates, word-break tables can contain comments delimited as follows:

>     /* this is a comment */

> For more examples, see the word-break tables that Progress provides in source-code form. They reside in the `DLC/prolang/convmap` directory and have the file extension `.wbt`.

> Note: Progress supplies a word-break table for each code page it supports.

> ##### Compiling word-break tables

> After you create or modify a word-break table, you must compile it with the `PROUTIL` utility. The syntax is as follows:

> Operating system | Syntax
> --- | ---
> UNIX Windows | proutil -C wbreak-compiler `src-file` `rule-num`

> ``src-file``

> The name of the word-break table file to be compiled.

> ``rule-num``

> A number between 1 and 255 inclusive that identifies this word-break table within your OpenEdge installation.

> The `PROUTIL` utility names the compiled version of the word-break table `proword`.`rule`–`num`. For example, if `rule`–`num` is 34, `PROUTIL` names the compiled version `proword.34.`

> ##### Associating compiled word-break tables with databases

> After you compile a word-break table, you must associate the compiled version with a database using the `PROUTIL` utility. The syntax is as follows:

> Operating system | Syntax
> --- | ---
> UNIX Windows | proutil `database` -C word-rules `rule-num`

> ``database``

> The name of the database.

> ``rule-num``

> The value of `rule`–`num` you specified when you compiled the word-break table.

> To associate the database with the default word-break rules, set `rule`–`num` to zero.

> Note: Setting `rule`–`num` to zero associates the database with the default word-break rules for the current code page. For more information on code pages, see the OpenEdge Development: Internationalizing Applications .

> ##### Word-break tables for double-byte and UTF-8 databases

> OpenEdge ships compiled word-break tables for double-byte and UTF-8 databases. The word-break tables are pre-applied to the appropriate empty databases. This means that word-indexing is available without any additional work for those databases.

> The compiled word-break tables are proword.245 through proword.254. The tables are in your `install-dir``/prolang/``language` directory or subdirectory with the corresponding empty database.

> If you already have a proword file that uses the same number as one of these new tables, you should change its number to a number less than 240. You can then apply the new word rule to your existing database. This change does not require an index build.

> ##### Rebuilding word indexes

> For word indexing to work as expected, the word-break table Progress uses to write the word indexes (to add, modify, or delete a record that contains a word index) and the word-break table Progress uses to read word indexes (to process a query that contains the `CONTAINS` operator) must be identical. To ensure this, when you associate the compiled version of a word-break table with a database, Progress writes cyclical redundancy check (CRC) values from the compiled word-break table into the database. When you connect to the database, Progress compares the CRC values in the database to the CRC value in the compiled version of the word-break table. If they do not match, Progress displays an error message and terminates the connection attempt.

> If a connection attempt fails and you want to avoid rebuilding the indexes, you can try associating the database with the default word-break rules.

> Note: This might invalidate the word indexes and require you to rebuild them anyway.

> To rebuild the indexes, you can use the `PROUTIL` utility with the `IDXBUILD` or `IDXFIX` qualifier.

> The syntax of `PROUTIL` with the `IDXBUILD` qualifier is:

> Operating system | Syntax
> --- | ---
> UNIX Windows | proutil`db-name` -C idxbuild [ all ] [ -T `dir-name` ] [ -TB `blocksize` ] [ -TM `n` ] [ -B `n` ]

> The syntax of `PROUTIL` with the `IDXFIX` qualifier is:

> Operating system | Syntax
> --- | ---
> UNIX Windows | proutil `db-name` -C idxfix

> For more information on the `PROUTIL` utility, see OpenEdge Data Management: Database Administration .

> ##### Providing access to the compiled word-break table

> To allow database servers and shared-memory clients to access the compiled version of the word-break table, it must reside either in the OpenEdge installation directory or in the location pointed to by the environment variable `PROWD` `rule`–`num`. For example, if the compiled word-break table has the name `proword.34` and resides in the `DLC/mydir/mysubdir` directory, set the environment variable `PROWD34` to `DLC/mydir/mysubdir/proword.34`.

> Note: Although the name of the compiled version of the word-break table has a dot, the name of the corresponding environment variable does not.

Copyright © 2005 Progress Software Corporation www.progress.com Voice: (781) 280-4000 Fax: (781) 280-4095