Status: Verified
SYMPTOM(s):
During crash recovery so many AI notes are generated that this fills up the current AI files to its limit
Can't switch to after-image extent it is full. (3773)
Database Server shutting down as a result of after-image extent switch failure. (5350)
Database fails to start as bi recovery exhausts available ai files
FACT(s) (Environment):
Database was previously emergency shut down
Database is ai archiver enabled
All Supported Operating Systems
Progress/OpenEdge Versions
OpenEdge Category: Database
All Supported Operating Systems
OpenEdge 10.1x
OpenEdge Category: Database
CAUSE:
Any database access that forces bi recovery phases (physical redo, Physical undo, Logical undo) also writes notes to the ai-extents. As such sufficient ai extent space is needed during crash recovery, or it will fail.
The aiarchiver neither archives ai extents when the database is shutting down nor when going through crash recovery at start up for example.
During shutdown, the Ai archiver shuts down immediately upon database shutdown. In other words, the archiver is not active while transactions are been either backed out or completed during shutdown. This means that AI extents will fill and switch when fixed size is reached without being archived during database shutdown.
During startup, the ai archiver is not initialised until crash recovery has completed. Any database access that triggers crash recovery, will initially fill the ai extents with notes during crash recovery. These include (but not exhaustive):
+ rfutil -C aimage new
+ rfutil -C aiarchive extent <path>\fullaiextent.an <path>\aiarcdir\aiextentbackupname.an
+ proutil -C truncate bi
+ proserve
Conversely, "rfutil -C aimage empty" CANNOT be run while aiarchiver is enabled
CAUSE:
Bug# OE00135777
FIX:
Upgrade to 10.2A.
Some workarounds are:
The recovery Options depend on which utilities are enabled alongside after-imaging. The consideration of sufficient bi file size for the bi notes is as important under these conditions, but not discussed in this Solution.
A. If the database is enabled for OpenEdge/Fathom Replication, ai extents of status "LOCKED" need to be considered, so please refer to Progress SolutionP124628, "Fathom Replication: source database goes down with errors 10601 3779 3773 on ai files"
B. If the database is NOT an OpenEdge/Fathom Replication enabled database and the ai archiver deamon has NOT been enabled, please refer to Progress Solution
P100481, "Database Server fails to start as a result of the error 5350"
C. If the database is NOT an OpenEdge/Fathom Replication enabled database and the ai archiver deamon HAS been enabled:
The following Options are offered for consideration in order of most to least drastic, where although the most-drastic solution is the fastest method to complete the bi crash recovery the replication re-baseline considerations should not be neglected.
OPTION#1: disable the ai archiver and after-imaging
1. disable both after-imaging and the ai archiver in one command.
$ rfutil dbname -C aimage end
After-image Extent Management has been disabled for the database. (13292)
After-image disabled. (846)
2. proceed with crash recovery:
$ proutil dbname -C truncate bi
3. Re-enable ai and ai archiving
$ rfutil dbname -C mark backedup
$ rfutil dbname -C aimage begin
$ rfutil dbname -C aiarchiver enable
4. Start the database
NOTE: A full online or offline probkup will need to be run in order to re-initialise the 'hotspare' baseline, since after-imaging was previously disabled.
OPTION#2: disable the ai archiver (temporarily) and manually copy and empty the FULL ai extents
1. disable the ai archiver
$ rfutil dbname -C aiarchiver disable
2. rfutil dbname -C aimage list >> ailist.out
3. OS copy the FULL extents noted from the output in STEP 2 above
4. mark the FULL extents as EMPTY
$ rfutil dbname -C aimage empty
5. Re-enable ai archiving
$ rfutil dbname -C mark backedup
$ rfutil dbname -C aiarchiver enable
5. Restart the database with the appropriate ai archiver database startup parameters: -aiarcdir -aiarcinterval -aistall -aibufs. The assumption is that there are now sufficient ai extents available to accomodate the crash recovery. Once crash recovery has completed and shared-memory intitialised, the ai archiver deamon will immediately archive and empty the FULL ai extents.
NOTE: There is no implicit need to re-initialise the 'hotspare' baseline, since after-imaging was NOT previously disabled. However this may still be advisable considering the cost of rolling forward all the ai notes that were generated during bi recovery.
OPTION#3: add more ai extents
NOTE: This Option is only available in Progress 9.1E or Open Edge 10.0B and later, (with the exception of OpenEdge 10.1B 10.1B01 and 10.1B02.
Please refer Progress Solution: P71887, "Unable to switch to new ai extent after adding a new ai extent to the database" )
1. Create a new file add_ai.st containing sufficient ai files to accomodate crash recovery. There is no absolute formulae for calculating this, except to suggest:
Note the number of ai extents switches and sizes previously needed during sh.utdown and subsequent attempts at recovery, then add at least twice this requirement assuming there are sufficient disk space availalbe for both the ai notes that need to be written from the bi notes generated during recovery.
Example: add_ai.st , 4 x 1GB fixed variable extents that will grow like variable ai extents to a maximum size of 1GB then switch during crash recovery
a . v 1000000
a . v 1000000
a . v 1000000
a . v 1000000
2. Add the new ai extents:
$ prostrct add dbname add_ai.st
3. Re-order the ai extents to ensure that the EMPTY ai extents immediately follow the current BUSY ai extent
$ echo y | prostrct recorder ai dbname
4. Refresh your structure file output
$ prostrct list dbname dbname.st
5. Restart the database with the appropriate ai archiver database startup parameters: -aiarcdir -aiarcinterval -aistall -aibufs. The assumption is that there are now sufficient ai extents available to accomodate the crash recovery. Once crash recovery has completed and shared-memory intitialised, the ai archiver deamon will immediately archive and empty the FULL ai extents..