Consultor Eletrônico

Status: Verified

SYMPTOM(s):

AppServer agents gradually increase CPU usage over time and system performance deteriorates

Right after the AppServer startup the performance is acceptable and CPU usage is normal

Performance degrades over time as agents are serving more requests

FACT(s) (Environment):

Restarting the AppServer broker or trimming AppServer agents helps to regain performance level
Progress 9.x
OpenEdge 10.x

CAUSE:

Application code does not release or deletes table buffers or temp-tables

FIX:

Review the application code executed on the AppServer agents for the
possible memory leaks associated with left over buffers, temp-tables, etc.
If the AppServer broker is running in the stateless operating mode, then
it is possible to use Activate and Deactivate procedures for logging additional
messages that can help to find out offending procedures.
Copy the following code and save it into the TTstats_act.p and TTstats_deact.p
which should be added into the AppServer's activate and deactivate procedures.
These procedures will output the messages in the AppServer server log file
which can be analyzed to establish where the leak is happening:

DEFINE VARIABLE cTables AS CHARACTER NO-UNDO.
DEFINE VARIABLE names AS CHARACTER NO-UNDO.
DEFINE VARIABLE cBuffers AS CHARACTER NO-UNDO.
DEFINE VARIABLE cBuffHdl AS CHARACTER NO-UNDO.
DEFINE VARIABLE cBuffTH AS CHARACTER NO-UNDO.
DEFINE VARIABLE cUniqTH AS CHARACTER NO-UNDO.
DEFINE VARIABLE cTempTxt AS CHARACTER NO-UNDO.
DEFINE VARIABLE a AS INTEGER NO-UNDO.
DEFINE VARIABLE numrec AS INTEGER NO-UNDO.
DEFINE VARIABLE hTT AS HANDLE NO-UNDO.
MESSAGE "TT Stats executed At: " + STRING(TIME, "HH:MM:SS").
RUN countSiblings(SESSION:FIRST-PROCEDURE, OUTPUT a).
MESSAGE "Persistent procedures: " a.
MESSAGE "Procedure files:~n" names .
RUN countSiblings(SESSION:FIRST-BUFFER, OUTPUT a).
MESSAGE "Buffers: " a.
MESSAGE "Buffer Names: ~n" cBuffers.
MESSAGE "Buffer Handles:~n" cBuffHdl.
MESSAGE "Buffer Table Handles:~n" cBuffTH.
MESSAGE "Buffer Table Names :~n" cTables.
DO a = 1 TO NUM-ENTRIES(cBuffTH):
cTempTxt = ENTRY(a, cBuffTh) .
IF LOOKUP (cTempTxt, cUniqTH) = 0 THEN cUniqTH = cUniqTH + cTempTxt + ",".
END.
cUniqTH = RIGHT-TRIM(cUniqTH,",") .
MESSAGE "Unique TT handles: " cUniqTH "~n Unique TT#: " NUM-ENTRIES(cUniqTH).
DO a = 1 TO NUM-ENTRIES(cUniqTH):
hTT = WIDGET-HANDLE(ENTRY(a,cUniqTH)).
RUN getTTstats (INPUT hTT, OUTPUT numrec).
MESSAGE "Temp-table:" hTT:NAME ":" STRING(hTT) " "
"Records # :" numrec .
END.
PROCEDURE countSiblings:
DEFINE INPUT PARAM pHandle AS HANDLE.
DEFINE OUTPUT PARAM pCount AS INTEGER.
DEFINE VARIABLE hTable AS HANDLE NO-UNDO.

ASSIGN pCount = 0
names = "".
DO WHILE VALID-HANDLE(pHandle):
pCount = pCount + 1.
IF pHandle:TYPE = "Buffer" THEN
DO:
cBuffers = cBuffers + pHandle:NAME + ",".
cBuffHdl = cBuffHdl + STRING(pHandle) + "," .
IF pHandle:TABLE-HANDLE <> ? THEN
DO:
cBuffTH = cBuffTH + STRING(pHandle:TABLE-HANDLE) + ",".
cTables = cTables + pHandle:TABLE + "," .
END.
END.
IF pHandle:TYPE = "Procedure" THEN
DO:
names = names + pHandle:FILE-NAME + "~n" .
END.> pHandle = pHandle:NEXT-SIBLING.
END.
cBuffers = RIGHT-TRIM(cBuffers,",").
cBuffHdl = RIGHT-TRIM(cBuffHdl,",").
cBuffTH = RIGHT-TRIM(cBuffTH,",").
cTables = RIGHT-TRIM(cTables,",").
END.
PROCEDURE getTTstats :
DEFINE INPUT PARAMETER hTT AS HANDLE NO-UNDO.
DEFINE OUTPUT PARAMETER nRecords AS INTEGER NO-UNDO.
DEFINE VARIABLE tablename AS CHARACTER NO-UNDO.
DEFINE VARIABLE qh AS HANDLE NO-UNDO.
DEFINE VARIABLE bh AS HANDLE NO-UNDO.
DEFINE VARIABLE lRet AS LOGICAL NO-UNDO.

tablename = hTT:NAME .
CREATE BUFFER bh FOR TABLE hTT NO-ERROR.

CREATE QUERY qh.
qh:SET-BUFFERS(bh).
qh:QUERY-PREPARE("FOR EACH " + tablename ).
nrecords = 0.
IF qh:QUERY-OPEN THEN
REPEAT:
qh:GET-NEXT().
nrecords = nrecords + 1.
IF qh:QUERY-OFF-END THEN
DO:
nrecords = nrecords - 1.
lRet = TRUE.
END.
IF lRet THEN LEAVE .
END.
qh:QUERY-CLOSE().
DELETE OBJECT qh.
DELETE OBJECT bh.
END PROCEDURE.
.