Status: Unverified
SYMPTOM(s):
Getting error 1153 when a JDBC client seems to experience some condition that results in _sqlsrv2 process dying.
BROKER 1: (1153) BROKER detects death of server 12099.
sql12099.trc file was created.
sql.trc file shows the following error:
PANIC - TPEUTIL hmh_itm_free !this PID 12099
JDBC client is making use of maxstatements >100
The parameter on the JDBC client side is:
<prop key="hibernate.c3p0.max_statements">100</prop> in the C3P0 connection pool.
Database server has a default value of 100 for -SQLStmtCache parameter.
SQLStmtCache parameter is set for all clients connecting to that specific _sqlsrv2 process.
maxStatements is JDBC's standard parameter for controlling statement pooling. maxStatements defines the total number PreparedStatements a DataSource will cache. The pool will destroy the least-recently-used PreparedStatement when it hits this limit.
From the client perspective it is the prepared statement close that fails and comes back with the "A Server Crash Likely" message.
Stack Trace from _sqlsrv2 process on HPUX Itanium machine:
_kill + 0x30
raise + 0x30
_abort + 0x190
_Z5panic11managerid_tPKcS1_ + 0x180
_Z12statusTPanici11managerid_tPKcS1_ + 0x290
_ZN8hm_hdl_t12hmh_itm_freeEPvi + 0x320
_ZN15valset_member_t8val_freeEv + 0x80
_ZN15valset_member_tD0Ev + 0x70
_ZN6vslist5clearEi + 0xf0
_ZN15sql_value_set_tD0Ev + 0x80
_ZN10stmtTmpl_tD0Ev + 0x300
_Z17ssm_reuse_attemptP9rds_env_tPKcS2_S2_P12sql_stmuid_thP10stmtTmpl_tPhPPcPP16ssm_stmtinfo_tag + 0x4c0
_Z12ssm_describeP9rds_env_tPKcP12sql_stmuid_thS2_P5sqldaP5sqlca + 0x3f0
sql_fn + 0x1500
_ZN16describe_reply_t15process_requestEv + 0x1c0
_ZN14server_reply_t13execute_replyEPPhPi + 0x2b0
_ZN16describe_reply_t7executeEPhPS0_PiS2_11sqlm_type_t + 0x70
_ZN12srv_nw_hdl_t13dsn_listen_10EiPvP10svr_args_t + 0xaa0
dmn_main + 0x300
_Z14run_sql_threadPv + 0x200
__pthread_bound_body + 0x190
FACT(s) (Environment):
Using Spring Hibernate c3p0 connection pooling for JDBC connection to a Progress database.
Issue does not occur if the JDBC clients do not make use of maxstatements parameter.
ia64 (Itanium)
OpenEdge 10.1C04 Service Pack
OpenEdge 10.1C04 Service Pack HotFix 23
OpenEdge Category: Database
CAUSE:
There have been defects in the area of re-preparing statements that have been previously evicted from the SQL statement cache. The SQL statement cache is handled on an LRU basic - old. Which means un-referenced statements get evicted when capacity is maxxed out and new statements arrive. If/when the old statement gets referenced, it is suppose to be re-prepared on the fly, but this was not always happening and resulted in the behavior being reported.
FIX:
Upgrade to 10.2A or 10.2B or later where several defects in this area have been addressed.
If you upgrade to 10.2X and you still experience the issue, you may be able to workaround it using the suggestion below. If you are able to readily reproduce the issue, please contact Customer Support at 781-280-4999 for further assistance. Please provide your Customer Support representative with the steps necessary to duplicate this issue so a defect can be logged to address your specific issue.
Workaround:
If you are unable to upgrade, you may be able to avoid this issue by changing the -SQLStmtCache on the broker to increase the SQL statement cache that the _sqlsrv2 process will use on behalf of the clients connected to that _sqlsrv2 process. Recommendations would be to set -SQLStmtCache 200. If the problem persists then increase this value to 400, 600, 800 or even 1000 depending upon which value allows you to avoid this issue. See solution: P37596, "Which Broker Should I place the -SQLStack, -SQLStmtCache, and -SQLCursor parameters?" for additional information on -SQLStmtCache.