Status: Verified
GOAL:
How to kill the AdminServer processes
GOAL:
How to kill the AdminServer broker on UNIX?
GOAL:
How to get a javacore against the AdminServer Broker ?
GOAL:
How to generate a thread dump against AdminServer processes?
GOAL:
How to force a stack against the AdminServer?
GOAL:
When the proadsv command stops responding (hangs) how to stop the AdminServer processes?
FACT(s) (Environment):
UNIX
Progress/OpenEdge Product Family
FIX:
While a forceful process shutdown (kill -9)" will get rid of the AdminServer processes, the associated stack trace for parallel investigations with the associated log files need to be gathered. It is very important that the correct process is terminated and the corresponding evidence generated is recorded. Please note simply issuing a kill against the process is not necessarily the best way forward in all cases and needs to be taken under cautious advisement.
Once a hang occurs, i.e. the proadsv command stops responding, the steps below should be used to ensure that the appropriate information is gathered. While undertaking these steps, be sure to keep track (by PID) which processes were killed.
Summary of steps:
1) find the AdminServer process. Find the path to the java runtime executable.
2.a) if the feature is available, ask the Java runtime to generate a Java core file.
2.b) ask the Java runtime executable to generate a UNIX core file.
2.c) if possible, get the stack trace from the core file
3) terminate the remaining processes
4) restart the AdminServer.
Detailed steps:
0) open a UNIX session as the system administrator (if possible), or as the user who has started the AdminServer process.
1) Find the AdminServer process, using the "ps" command.
For example, on AIX/Solaris:
# ps -ef | grep java | grep AdminServer
On GNU/Linux:
# ps auxw | grep java | grep AdminServer
Take note of the full path to the java executable for later on. (**) Example: "/usr/java131/bin/java"
2) Ask the process to tell you as much as possible about its current state.
2.a) Steps differ depending on the platform.
For AIX: when you're sure you have the correct process, send it a SIGFPE signal (signal 8). Use: kill -8 <PID>
This asks the java layer (javacore) to create a dump of its current state in a file named "javacore<number>" in the working directory. Save this evidence.
A listing of javacore's created can be referenced in "/tmp/javacore_locations". Note that this feature is not available on all UNIX implementations.
For other systems using the Sun Java virtual machine): send the process a SIGQUIT signal (signal 3). Use: kill -3 <PID>
This asks the java layer to create a Java stack trace on its console, which is redirected to the AdminServer log file, or a file named admserv.exp in the working directory. Save this evidence.
2.b) If the process did not terminate with the previous step, send the signal SIGABRT to the process, to get the information from the c-layer. Use: kill -6 <PID>
This will result in a "core" file in the working dir. Save it. Note that a number of files named protrace.<pid> could also be created (from the running Progress sessions started by AdminServer). Save this evidence as well.
2.c) Use a debugger to get a stack trace from the "core" file. You will need the complete path to the java executable (extracted above).
For example if dbx is available, using the information saved above (**) :
$ dbx /usr/java131/bin/java core | tee core.out
[dbx]where
.
.[dbx]quit
If dbx is not available but "gdb" is, use gdb instead.
3) Some processes may have not terminated gracefully with the previous steps. You need do remove any remaining process. For this first send SIGTERM (15) to the remaining processes which are not responding (kill -15 ...). After a delay, if this is not sufficient use SIGKILL (kill -9 ...).
Then if on AIX, run "/usr/sbin/slibclean" which removes any currently unused modules in kernel and library memory.
4) Restart the AdminServer .