Consultor Eletrônico

Status: Verified

SYMPTOM(s):

An integer in a calculation casts the whole calculation to type integer

Calculation is not of datatype integer

If the result of calculation exceeds the maximum possible integer value, an incorrect large negative value is produced.

There is no error trap to indicate the integer overflow condition.

FACT(s) (Environment):

All Supported Operating Systems
Progress/OpenEdge Product Family

CAUSE:

When two operands in a calculation are of type integer, the calculation is carried out with integer arithmetic and if the product of this increment in the calculation overflows the 2,147,483,648 integer boundary, then the result of the calculation will reflect this state as there is no error trapping in this case.

Conversly, if one of these operands are decimal, then the operation will be carried out with decimal arithmetic where there is no overflow into the -2 billions and the calculation returns the expected result. If the decimal overflow is reached, an error message: "Decimal number is too large (536) is returned and no result is calculated from the equation.

FIX:

Taking the integer value OUT of the equation solves the problem as the calculation then works correctly because the calculation is then cast to type decimal.

For example:

DEF VAR v-i AS INTEGER INITIAL 2147483647 NO-UNDO.
DEF VAR v-wrong AS DECIMAL INITIAL 0.0 FORMAT "$->>,>>>,>>>,>>9.99" LABEL "Wrong1" NO-UNDO.
DEF VAR v-correct AS DECIMAL INITIAL 0.0 FORMAT "$->>,>>>,>>>,>>9.99" LABEL "Correct" NO-UNDO.

/* both operands "(v-i + 1)" are integers, so the calculation will be carried out with integer arithmetic */
v-wrong = ((v-i + 1) * 1 / 1) - 1.
DISPLAY v-wrong WITH FRAME f-frame 10 DOWN 1 COLUMN.

/* one of the operands "(v-i + 1.0)" is decimal, so the operation will be carried out with decimal arithmetic */
v-correct = ((v-i + 1.0) * 1 / 1) - 1.
DISPLAY v-correct WITH FRAME f-frame.