Consultor Eletrônico

Status: Unverified

SYMPTOM(s):

Passing input parameters of type 'double' from a Progress application to a DLL (shared object) results in incorrect precision on IBM AIX

Parameters undergo a small but non-negligible change during transfer


For example, the value 0.9 (of type DECIMAL) is passed as input parameter (of type 'double') to a function in the DLL; the result is:

0.89999997615814208984

The value should be (+/-):

0.90000000000000002000

FACT(s) (Environment):

Problem does not occur on IBM AIX when the DLL is called by a C++ program
Problem does not occur on Windows
OpenEdge Category: Language (4GL/ABL)
OpenEdge 10.1x
OpenEdge 10.2A
OpenEdge 10.2B
IBM AIX

CAUSE:

Bug# OE00193957

CAUSE:

IBM AIX 64-bit uses avcall 1.8 to build the parameters for the shared object call. It defines av_double() as:

#define av_double(LIST,VAL) \
(++(LIST).aptr > __av_eptr(LIST) \
? -1 : (((double*)(LIST).aptr)[-1] = (double)(VAL), \
(LIST).faptr < &(LIST).fargs[13] && \
(*(LIST).faptr++ = (float)(VAL)), \
0))

There is a cast of VAL to float which causes the problem. Progress executes the FRSP instruction. Per the doc, the FRSP instruction "rounds a 64-bit, double precision floating-point operand to single precision". 0.9 is initially stored as 0x3feccccccccccccd in a floating point register (0.90000000000000002), but the FRSP instruction changes this to 0x3fecccccc0000000 (0.89999997615814209).

The version of avcall differs between platforms, which is why some platforms might not suffer this. Windows (32-bit) does not use avcall at all, which is why the result is different.

FIX:

Upgrade to 10.2A03