Kbase P119170: What is buffer pool and Memory Limitation
| Autor |
Progress Software Corporation - Progress |
| Acesso |
Público |
| Publicação |
08/09/2008 |
|
Status: Verified
GOAL:
What is buffer pool
GOAL:
Progress RDBMS Buffer Pool Size Limitations
GOAL:
How the buffer pool size is determined
GOAL:
How memory is allocated for the buffer pool
FACT(s) (Environment):
All Supported Operating Systems
FIX:
Progress RDBMS Buffer Pool Size Limitations
There are a number of factors that affect the limitations on the buffer pool size in modern operating systems.
What the buffer pool is
The buffer pool is used to keep copies of popular database blocks in memory to minimize disk i/o. There are a number of complex data structures involved, but the most important are the buffers themselves and the buffer descriptors (sometimes called buffer headers).
Each buffer contains a copy of one database block and is the same size as a database block. For each buffer, there is a buffer descriptor that describes what is in the buffer and some information about it (updated or not, which database block it is, whether the buffer is locked or not, and so forth. The buffer descriptors are around 150 bytes long.
How the buffer pool size is determined
These two structures are typically the database manager's largest consumers of memory. There is a server startup parameter, -B, that is used to specify how many buffers (NOT how much memory to use) and how many buffer headers should be kept in memory. So the number of buffers specified for -B determines how much memory the buffer pool requires.
For example, let's say we have a database with 1024 byte blocks and you specify a -B of 1000. We will require a buffer pool of approximately 1,180,000 bytes. If the database block size were 8192 bytes, then we would need approximately 8,550,000 bytes.
How memory is allocated for the buffer pool
There are two independent methods for allocating memory, depending on whether you are using the database in single or multi user mode.
In single user mode, the database is accessed exclusively by a single program or process. This process could be a Progress session running a 4GL application, or it could be one of the various database utilities, like index rebuild or offline backup. Regardless, that process contains an instance of the database manager and it allocates memory for everything including the buffer pool from its own address space using the host operating system's memory allocator. On Unix the standard method is by calling the malloc() library function.
In multi user mode, most of the database manager's data structures including the buffer pool are kept in shared memory. It is the fact that the database manager's data structures are kept in shared memory that makes it possible for multiple process to operate efficiently on the same database. Depending on the operating system, shared memory is allocated in one lump or several segments. The database manager calculates how much memory is required and then creates the appropriate amount of shared memory to hold everything. Space fhe individual data structures are allocated from the shared memory by a special memory allocator internal to the database manager, not by operating system functions
Address spaces and memory allocation
All of the memory accessible to a program is called its "address space". Everything it touches has to be somewhere in the address space. Aside from the buffer pool, there are a host of other things that also go in the address space. Things like:
* The code for the program itself (the "text segment" on Unix)
* Code in shared libraries or dynamic link libraries ("dll's") the program uses
* The program stack
* Thread stacks
* Constant or static data
* Global variables
* The "heap", a pool of private memory from which space for various temporary data structures, like Progress widgets, r-code buffers, r.ecord buffers, etc. is taken from.
* Shared memory segments
* File buffers for open streams
* Memory mapped files
* etc, etc, etc,
* The code and data structures of the operating system. Even though a program can't normally touch those directly, at lesat on good operating systems (Windows 95 is not a "good" operating system, they have to go somewhere
The various operating systems all have different strategies for determining where to put all these things. Once they have been created, they usually can't be moved somewhere else without going to an awful lot of trouble and expense. And the maximum size of the address space is finite. Some of the limitations on the sizes of things arise from the operating system's strategy for where to put things and others from the total amount of stuff that has to go somewhere.
Every datum in memory has an address that specifies its location in memory. The size of the address is limited, usually by the underlying cpu architecture. A data element that contains an address of something else is called a "pointer". Depending on the processor type, a pointer is 16, 32, or 64 bits. Other sizes were once common but no longer.
Limitations on the size of the buffer pool
One of the limitations on the size of the buffer pool is the number of buffers Progress allows. This varies by release and to some extent by operating system. The table below lists limits on the value of -B:
Upper limits
Version 6.2 and before 32,000
Version 6.3 500,000
Version 7.3D 500,000
Version 7.3E 125,000,000
Version 7.3F 125,000,000
Version 8.1A 125,000,000
Version 8.2A 125,000,000
Version 8.3 125,000,000
Version 9.0B 64-bit 125,000,000
Version 9.0B 32-bit 500,000
Version 9.1 125,000,000
Note that 500,000 8k buffers is about 4GB, and 125,000,000 8k buffer is about 8,000 GB.
A note on the (irrelevant) lower limit:
On all systems, silently enforced, is 10. If you say -B 1, you get 12. You always get at least 12, and 2 more than you ask for. The two extra are for the master block and the first index anchor table block, which are fixed in memory in the buffer pool. Except that version 9 doesn't have index anchor tables.
Back to the story on upper limits:
The fact that the upper bound on -B is some particular value is not the only limitation. The other limitations come from some other things: the maximum size of the address space and how the operating system uses it, and the maximum size and maximum number of shared memory segments that can exist.
With a 32-bit address space we can address 4GB of memory but we cannot use all of it for shared memory. The limit on total shared memory varies by operating system and ranges from around 1.25 GB to a little over 3 GB.
The database manager has an upper limit of 16 MB, 128 MB, or 1 GB for shared memory segment size, depending on the operating system.
AIX 128 MB (134,217,728 bytes)
Win NT 128 MB (134,217,728 bytes)
Digital Unix 1 GB (1,073,741,824 bytes)
Others 16 MB (16,777,216 bytes)
AIX allows a process to attach to at most 10 (11 in the latest version) shared memory segments. There is a way provided in AIX 4.3 to overcome this limitation by the use of an EXTSHM environment variable.
HP-UX 10.x and 11 have an upper limit of ~1.75 GB of shared memory.
There is a way to raise this limit to 2.75 GB, but to do so requires linking the executables in a special way before we can take advantage of it.
Another important thing to consider is the following: When you start a server for the .database, that server accesses only one database. A client has the capability to connect to multiple databases and a self-serving client does so with the server code bound into the same process as the client code. Self-serving clients have to attach to all the shared memory of each database they connect to. So now multiple database's data structures all have to go into the same address space and this imposes another limit..