Kbase 19310: Problems Starting WebSpeed Agents on HP-UX 11.0
| Autor |
Progress Software Corporation - Progress |
| Acesso |
Público |
| Publicação |
2/7/2011 |
|
Status: Verified
SYMPTOM(s):
WebSpeed agents fail to start
AppServers fail to start
Receiving a Java Out of Memory error
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at com.progress.ubroker.broker.ubThreadPool.startThread(ubThreadPool.java:1810)
java.lang.OutOfMemoryError: max threads limit being reached in VM
at java.security.AccessController.doPrivileged(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:77)
at java.lang.Runtime.execInternal(Native Method)
Maximum number of threads per process is reached
Errors are present in the Broker log file
And/or many client's requests at the time of error
FACT(s) (Environment):
Starting many agents/AppServers per Broker
HP-UX 11 32-bit
HP-UX 11 64-bit
Progress 9.x
WebSpeed 3.x
CAUSE:
The error indicates a system limitation in regards to how many threads a specific process can handle. According to Hewlett Packard, the default values for HP-UX 11.0 are set too low for most Java applications.
The WebSpeed, AppServer brokers, and other other unified brokers in 9.x / 3.x are a Java program, and run within a Java Virtual Machine (JVM). The broker is multi-threaded for efficiency and scalability. These threads are all initiated from the one JVM process.
Consider the following about the ubroker thread allocation:
Description # of Threads
----------- ------------
Main thread 1
Well-known port Listener 1
autoTrim 1
RMI thread 1
N-many AppServers 3 X N
M-many clients 1 x M
This means that in the default configuration you start 5 AppServers, so 19 threads are required without counting any client connections. If you want your broker to support 20 AppServers, then the broker needs 65 threads before even a single client connects!
In a real system, you must also include the "M number of client threads" into the calculation. This number is based on how many concurrent client's requests come in. Every time a client attempts to connect to the AppServer broker, for instance, a client thread is spawned up to the maximum number of client instances (maxClientInstance, by default, it is set to 512). When a client tries to connect to a server, the broker spawns a client thread which will either get a connection to a server, if one is available, or wait for a while until one becomes available (or timeout in case it can't get to a server).
NOTE: the system might require a thread or two for its own purposes, however these numbers are close enough to illustrate the point.
To support these numbers of threads, the JVM that is required from Progress on HP-UX 11.0 requires a kernel setting for the maximum number of threads a process can start. By default, this value is too low to run a large number of agents on a single broker. It is possible to start another broker, and have more agents on this broker, but you will not be able to add more agents to the initial broker.
FIX:
Increase the max_thread_proc kernel parameter, nkthread, nfile, maxfiles, maxfiles_lim, and ncallout.
See a UNIX system administrator for further details. This is outlined in the technical document from HP. The kernel must be rebuilt and the system rebooted for the changes to take effect. This is an issue with the JVM supplied by HP; it is not a Progress problem.
http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1616,00.html
The value of the max_thread_proc must be set higher than the expected maximum number of simultaneously active threads for the application. max_thread_proc is the maximum number of threads allowed in each process. The minimum value (and default) is 64. The maximum value is the value of nkthread.
nkthread is the total number of kernel threads available in the system. This parameter is similar to the nproc tunable except that it defines the limit for the total number of kernel threads able to run simultaneously in the system.
The value of nkthread must be greater than nproc. The default is approximately twice that of nproc. The maximum is 30000. The suggested value of nkthread is 2*max_thread_proc.
Keep in mind that nkthread limit should be based on what you set the max_thread_proc.
Reducing maxClientInstance to a reasonable number instead of setting it to default value of 512 would also help.