Status: Verified
GOAL:
How to analyze a process crash or hang when the cause is unknown
GOAL:
How to troubleshoot crashing or hanging _progress, _proapsv, _mprosrv or other Progress tools
GOAL:
How to use ADPlus, a debugger and other tools to troubleshoot hangs or crashes
FACT(s) (Environment):
Progress/OpenEdge Versions
All Supported Operating Systems
FIX:
Prologue: this knowledge base entry pertains to process crashes or hangs, not ABL errors nor application hangs that are due to the process running a loop in the ABL. These latter issues need to be investigated using regular 4GL debugging techniques. See the Debugging and Troubleshooting Guide for more details. Also, this solution is not intended to be a replacement to a thorough debugging and troubleshooting methodology. Whenever possible, crash or hang issues should be submitted to Progress Technical Support as a simple reproducible case that allows Technical Support to reproduce the issue in-house. When a Progress process is found crashing or hanging, the general approach to diagnose the issue is as follows: - try and isolate the specific feature or behavior of the product that is triggering the issue. For this purpose the debugging output options for logging can be useful. See the related solutions linked below for details. Then try to find a matching solution in the knowledge base.
- if the issue cannot be isolated, or no matching solution is found, contact Progress Technical Support for further investigation. However, in order to optimize the time needed for the investigation, the information below could be used to prepare the case before submission. Rationale: During the troubleshooting process, it may be that the debugging output available through standard product features is not sufficient to provide an understanding about the cause of the issue. Should this case arise, Progress Technical Support may ask to investigate the current internal state of the running process at the moment it crashes or hangs. This requires additional details from the running environment that can be gathered before the issue is submitted to Technical Support. The following details can be gathered: - the name (and path) of the executable file running the process being considered. It need to be an executable provided by Progress (i.e. the information below does not apply to 3rd-party executables).
- the exact version number and service pack / patch level of the Progress executable that crashes or hangs. In case of doubt, collect the exact size (in bytes) and time stamp of the executable file.
- a stack trace of the executing process at the point in time where the issue arises. If the process seems hanging, multiple stack traces generated at short intervals can be useful to determine whether the process is waiting or running in a loop. To obtain a stack trace:
- if the process is crashing, and a protrace file is generated, and the protrace file contains a list of memory addresses and function names, then it may contain a valid stack trace and can be submitted to Technical Support.
- in other cases, consider the two sections below (depending on the operating system). Consider contacting Progress Technical Support before attempting these steps. In some situations (especially for some executables) these steps will bring no valuable information. On Unix-related operating systems (Linux included): - if the process is _progress, _proapsv or _mprosrv, and it appears to be hanging, then proceed as follows:
1. send the signal SIGUSR1 to the hanging process multiple times (at 1-second intervals). This should cause no other side-effect than forcing the generation of a stack trace. The process should continue running afterwards.
2. If a protrace file is generated and contains a list of memory addresses and function names, then it may contain an appropriate stack trace and can be submitted to Technical Support. - if the process is Java-based (including _sqlsrv2), try to obtain a Java exception log or .stack trace. For this purpose refer to your Java VM documentation or contact Progress Technical Support. - if no protrace file is available, the process is not Java-based, and a debugger (dbx, gdb or other) is available on the system, then proceed as follows:
1. if the process is still running, determine the process ID (PID)
2. if the process has crashed and a « core » file has been generated, collect the core file
3. attach the debugger to the process. For example:
dbx <path to executable> <PID or core file>
gdb <path to executable> <PID or core file>
4. at the debugger prompt enter the command to generate a stack trace (with gdb and dbx the command is called "where") and save the output. This can be submitted to Technical Support.
It is essential that the <path to executable> used points to the exact same executable running the process that is seen hanging or crashing. - if no debugger is available, then proceed as follows:
1. ensure that a « core » file is generated. If the process is hanging, send the signal SIGABRT to force the generation of a core file. Note that the generation of a core file may be disabled; use the command « ulimit -c » to get or set the current settings for core file generation.
2. contact Progress Technical Support for further instructions. On Microsoft Windows platforms: - if the process is Java-based (including _sqlsrv2.exe), try to obtain a Java exception log or stack trace. For this purpose refer to your Java VM documentation or contact Progress Technical Support. - if no protrace file is available, the process is not Java-based, consider using the tool ADPlus from Microsoft to get additional information about the issue, as per the suggestions below Prerequisites: - If the issue is a crash, then the crash must be reproducible. No additional information can be gathered using this method after the crash has happened.
- ADPlus must be installed. See solution P116692, What is ADPlus and where can it be found?
- The instructions below refer to the ADPlus executable file adplus.vbs; newer versions of ADPlus may use adplus.exe instead. For hanging processes: 1. obtain the process ID of the process (using Windows' Task Manager) 2. use ADPlus to generate a memory dump and stack trace of the hanging process:
adplus.vbs -hang -p <PID> 3. collect the generated log file and memory dump. As a first step send the log file to Progress Technical Support. The Technical Support Engineer may request the memory dump at a later time so it should be archived. For crashing processes, when the crash does not happen immediately on startup: 1. ensure the process is started 2. if the crash is known to happen in a specific process:
2.a. obtain the process ID of the process (using Task Manager)
2.b. attach Adplus to the process:
adplus.vbs -crash -p <PID> 3. if the crash can happen in multiple running processes based on the same executable file:
3.a. determine the name of the running image (executable name)
3.b. attach Adplus to the family of processes:
adplus.vbs -crash -pn <process name>
3.c. trigger the crash or wait for it to happen 4. after the crash happens, collect the generated log file and memory dump. As a first step send the log file to Progress Technical Support. .The memory dump should be archived in case the Technical Support Engineer requests it at a later time. For crashing processes, when the crash happens immediately on startup: 1. start the process with ADplus so that it is immediately attached:
adplus.vbs -crash -sc "<path to executable> <command line parameters...>" 2. after the crash happens, collect the generated log file and memory dump. As a first step send the log file to Progress Technical Support. The memory dump should be archived in case the Technical Support Engineer requests it at a later time..