Consultor Eletrônico

Status: Verified

GOAL:

Diagnosing codepage related problems

GOAL:

How to diagnose CodePages problems?

GOAL:

Troobleshooting CodePages problems?

GOAL:

How to troobleshoot CodePages problems?

FIX:

Codepage related problems can be extremely complex to diagnose because of the fact that there are many possible points of failure. If you cannot view non-USASCII characters correctly you cannot just assume that checking and correcting one error makes your display correct. In many cases there are two or more things unknown or wrong in the setup and they interfere with each other. Before starting this investigation you should be familiar with the Progress character processing as documented in the manuals (V6: Programming Handbook chapter 2, V7 and V8 and higher: System Administration Guide, appendix A). You should
also know what codepages your devices (Terminals, printers etc) use. To keep this notebook reasonably small, no instructions are provided on how to correct the problems you encounter. That information can be found in the Progress documentation and in the documentation supplied with the relevant third party products.

As in all troubleshooting, try to narrow down the problem by eliminating possible causes. If something has changed in the system configuration, suspect that change first. If you only have problems with printing and your terminal or GUI display is ok, you should suspect the print settings. Note that this still does
not imply that the rest of the system configuration is correct. When dealing with codepage problems you just cannot assume that what you see really is what you see. There is always the possibility that the terminal you are using to rectify the situation is misconfigured. In general, try to check the actual numeric codes of characters if possible.

The instructions in this notebook assume that you have not modified the protermcap or convmap.cp files. If you have special versions of these files, you should, if possible try testing with original files supplied with the product. This is of course impossible if you are using a codepage or collation that is not shipped with the Progress product. In this case you should first check the validity of your modifications.

Verify that the Progress session startup parameters on the client and server are valid. If you are using special startup scripts, make sure all parameters are correctly passed to the executables. Also check for conflicting parameters in $DLC/startup.pf (This file is read by all executables).

With any codepage and/or collation problem one of the things to check is the database itself. If your database contains incorrect or corrupted data it is virtually impossible to fix by quickly changing other settings. Start a client session with parameters "-cpstream undefined -cpinternal undefined", connect to your database and run the following tests. Note: Running with -cpinternal undefined is crucial to defeat any automatic conversions. This procedure works only with one database connected. For multiple databases you have to do it
separately.

find first _db.
display session:charset format "x(10)" label "Internal codepage"
session:stream format "x(10)" label "Stream codepage"
_db._db-xl-name format "x(10)" label "Db codepage"
_db._db-coll-name format "x(10)" label "Db collation".

Verify that the values displayed for database codepage and collation are correct. Also, if you don't get "undefined" displayed for your internal and stream codepages, you should check your Progress startup routines. The next thing is to find some extended characters and display their respective numeric character codes. You have to find a field that contains extended characters and use the following procedure to display the value (Example against customer.name field in isports database provides value 228).

find customer where cust-num = 7.
display customer.name
asc(substring(customer.name,11,1)).

Find multiple fields in different tables to make sure all tables are using the same codepage and all characters that should be the same really are the same. The values should match the values listed in .codepage table for the database codepage. Note that this procedure displays the values in decimal and most codepage tables are done in hexadecimal.

The internal codepage is used for all 4GL comparisions and sorting which does not use a database index. This codepage and collation is independent from the database and can be different. GUI input/output uses internal codepage and cannot be converted. If in any doubt, override the defaults by explicitly specifying
the -cpinternal and -cpcoll parameters.

Check for hard-coding certain codepage values in comparisions or assuming that the comparisions are always in ASCII order. A common mistake is to assume that CHR(255) > "Z". This works with ibm850 codepage but in iso8859-1 CHR(255) is ydiaeresis which (in "basic" collation) sorts like y. In general the ASCII value of a character does not tell you anything about the collating weight of that character. Collating weights are language-dependent and for one codepage there are several collating sequences. To determine a collating weight of a character (i.e. how it sorts) in a particular collation, refer to $DLC/prolang/convmap.dat file.

With GUI screen codepage issues be sure to check what codepage your windowing system is actually using. This must match the Progress -cpinternal setting. Also try with different fonts and check that the font encoding is correct. Many X11/Motif distributions contain fonts in different codepages and all
MS-Windows fonts do not contain the same characters.

The definition stream I/O applies to almost anything that is not GUI screen or database I/O (files, printers etc). Character terminal I/O is a subcategory of stream I/O as far as codepages are concerned. Procedure source code is read by compiler and converted to internal codepage according to -cpstream setting.

If you have codepage problems with terminals, check the codepage the terminal is actually using. Many terminals and emulators have numerous language-specific setup items and their usage and impact is rarely documented well in the manuals. In most cases these can be overridden with software commands (Escape sequences) so it is necessary to check what the settings are when you are running Progress. It can well turn out that your login scripts or Progress startup terminal initialization changes these settings. Also, some operating systems have terminal line codepage mappings that can sometimes really fool you if you are not aware of them. SCO Unix is one of those operating systems. To be sure that the characters you enter at the keyboard are correct and the characters displayed are correct, use the following procedure (Again, start Progress with -cpinternal undefined -cpstream undefined).

define variable v-ch as character format "x".
update v-ch.
display asc(v-ch).

When you run this procedure, enter one extended alphabet at a time and compare its numeric value to a codepage table. It is good to check this for every character, upper- and lowercase. You should get ascii values that correspond to the values listed in codepage table. If they don't match, you can either
try to find a codepage that they match or try to find what piece of your system is doing a conversion "behind your back". Some (Particularly older) hardware and software implementations are notoriously bad at stripping the 8th bit of a data stream, effectively corrupting the data.

Input and output file codepages can be easily checked using OS tools. In Unix, use "od -x" to view hexadecimal character codes. In DOS/Windows you can easily check if a file is either in the correct DOS or Windows codepage: Just load it into DOS Edit or Windows Notepad. You can use any editor you wish but make sure it does not do codepage conversion when it loads the file. The codepage of these editors of course depends on your DOS/Windows international settings and you should know that in advance. Printing codepage in DOS/Windows must genera.lly be in the OS codepage and in Unix you can determine that by printing a test file with normal OS tools.

===================================
Progress System Administration Guide, Appendix A
Developing International Software (Nadine Kano,Microsoft Press)
System software and hardware manuals.