Kbase P135466: Database crashes with bkio read errors weekly due to insufficient resources on Windows
| Autor |
Progress Software Corporation - Progress |
| Acesso |
Público |
| Publicação |
5/17/2011 |
|
Status: Verified
SYMPTOM(s):
<function>:Insufficient disk space during <system call>, fd <file descriptor>, len <bytes>, offset <bytes>, file <file-name>. (9450)
Getting bkioread errors regarding insufficient disk space when doing a progress backup.
Errors are being reported on the fixed extents for certain data areas in the database.
Database crashes weekly due to insufficient resources on Windows 2003
Clients and also the backup process hit the same error messages. Sometimes they terminate abnormally, sometimes they successfully complete after retrying.
SYSTEM ERROR: <function>: Bad file descriptor was used during <system call>, fd <file descriptor>, len <bytes>, offset <bytes>, file <file-name>. (9446)
SYSTEM ERROR: read wrong dbkey at offset <offset> in file <file> found <dbkey>, expected <dbkey>, retrying. area <number> (9445)
Corrupt block detected when reading from database. (4229)
<func-name>: Error occurred in area <num>, block number: <num>, extent<name>: . (10560)
Writing block <num> to log file. Please save and send the log file to Progress Software Corp. for investigation. (10561)
SYSTEM ERROR: Wrong dbkey in block. Found <dbkey>, should be <dbkey2> in area <num>. (1124)
Begin ABNORMAL shutdown code (2249)
FACT(s) (Environment):
Doing an off line backup of the database using the probkup command.
Event Viewer log sometimes states the following error message:
Windows cannot load the user's profile but has logged you on with the default profile for the system.
DETAIL - Insufficient system resources exist to complete the requested service.
For more information, see Help and Support Center at
http://go.microsoft.com/fwlink/events.asp
Windows
OpenEdge 10.1x
OpenEdge 10.2x
CAUSE:
Bug #OE00181771
CAUSE:
There is not enough memory available to the Windows Paged Pool when the server is very busy and demand for memory is high.
FIX:
Increase the Windows PagedPoolSize value to allow a larger paged pool. A setting of 0xFFFFFFFF will allocate the maximum Paged Pool in lieu of other resources to the computer.
Suggestions from Microsoft:
You will need to enter 0xFFFFFFFF for the PagedPoolSize which is the maximum.
Default PoolUsageMaximum is 80. So Memory Manager trims pool memory at 80%.
Below you will see an article that informs about pool resources on x64:
Upgrading to a 64 bit platform will give a lot more pool resources (up to 128 GB instead of 256MB Non-paged and 470MB Paged Pool)
294418 Comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003
http://support.microsoft.com/default.aspx?scid=kb;EN-US;294418
We already see in the Perfmon that the system was critically low on Pool Paged memory with less than 20% available, it did not yet generate a 2020 / 2021 event
But instead your customer gets a error 1450, the article below may be helpful.
317249 How to troubleshoot Event ID 2021 and Event ID 2022
http://support.microsoft.com/default.aspx?scid=kb;EN-US;317249
Extract:
Other components of the operating system may not work and may generate error messages that report a status code of 1450 in the data section of their event log message. That is, "Insufficient System Resources." These events may be found in the System event log or in the Application event log. These messages may apply to the issue that is described in this article only if the underlying event was a connection to the server service. However, this fact is not easily determined. For example, there is Event ID 1055 that is generated by CLUSSVC. This event is from the cluster service that usually reports a failed connection to the server service.
Based on the symptoms I have seen in this case I would encourage the customer to move to x64. It is not possible for me now to see if it will delay the issue, however it is not very likely this will happen. Only with a memory dump we are able to get more grip on what is actually happening. Based on the Perfmon only, a x64 OS will relief the system. Fine tuning the current system is possible but may be a lengthy process.