Kbase P123873: Prostrct addonline hangs if all AI extents are LOCKED and busy extent is full
| Autor |
Progress Software Corporation - Progress |
| Acesso |
Público |
| Publicação |
6/1/2010 |
|
Status: Verified
SYMPTOM(s):
prostrct addonline hangs if all AI extents are LOCKED and busy extent is full
Database is stalled due to using -aistall database startup parameter
Database activity is stalled until an AI extent becomes available. (14218)
prostrct add FAILED. (12867)
FACT(s) (Environment):
prostrct addonline <dbname> <dbname.st>
No errors in database log file
OpenEdge 10.1B
All Supported Operating Systems
CAUSE:
Bug# OE00149699
CAUSE:
This problem is not caused by Replication, but Replication does make it more difficult to empty extents because the extent may not be replicated to the target. In which case the extents are LOCKED status as opposed to FULL and therefore cannot be EMPTY'd.
When the database is in an AI stall (-aistall in use), a process is attempting to write a note to the current AI extent. This process remains blocked until the AI stall has been resolved by emptiing an extent. If the AI stall is never resolved, the addonline will block waiting for the stall to be resolved - effectively hanging the addonline process. The addonline command depends on the same resources as the process that is blocked behind the AI stall and 2 processes cannot be using the same resources at the same time. So the addonline ends up blocking, waiting for the resources and latches that are in use and held by the process blocked by the AI stall.
FIX:
Fixed in 10.1B03, where prostrct addonline is prevented from waiting by first checking for the aistalled state and exiting gracefully if it is not be able to complete the requested utility.
Workaround1:
If any AI areas are FULL, archive and empty the areas then retry the prostrct addonline command.
Workaround2a:
Prior to OpenEdge 10.1B03, if OpenEdge Replication is enabled and all extents are LOCKED there is no way to add additional AI areas. In which case OpenEdge Replication online will need to be disabled online, then archive and empty the FULL areas:
dsrutil <dbname> -C disablesitereplication source
rfutil <dbname> -C aimage extent empty
After disabling replication, re-create the replication target database and re-enable replication when next convenient.
Please note the Solution references below for related issues in this area: P123871 P123872
Workaround2b:
OpenEdge 10.1B03 +
1. Disable the ai archiver deamon time switch if in place:
rfutil source -C aiarchiver setinterval 0
or stop the cronjob that manages ai switching. Once the ai file issues have been resolved these utilities can be restarted.
2a. The ai extents on the source are LOCKED in the replication enviroment most likely because replication has stopped.
To verify this, run the dsrutil source -C monitor utility, it is not running if it returns:
Cannot connect to replication shared memory. Status = -1
Restart replication by:
a. restart the target database:
$ proshut target -by (NOTE: do not force shutdown of the target database)
$ proserve target -DBService replagent [+parameters]
b. restart the replication server on the source database:
$ dbutil source -C restart server
monitor the status of these with the "dsrutil source -C monitor" utility.
When the source and target databases are synchronised the LOCKED extents will become FULL. These FULL extents will become available for ai notes once they are backed up (for roll forward plans) then emptied with the 'rfutil source -C aimage empty' utility or by the ai management daemon when restarted.
2b. If it is not possible to restart replication, add new ai extents offline:
$ proshut source -by
$ prostrct add source newaifiles.st
where newaifiles.st contains the structure definitions of the new ai files, eg:
a . v 20000
a . v 20000
a . v 20000
$ prostrct reorder ai source
$ prostrct list source # to update the .st file
$ rfutil source -C aimage list # to check that there are now EMPTY extents after the current BUSY extent.
Finally re-start the source database.
NOTE: Sufficient ai files need to be added to accomodate the time needed to restore the replication baseline or to resolve the source <> target replication. Otherwise, consider disabling replication until such time as the replication environment is restored.