Using Granularity Hints to Improve Performance ---------------------------------------------- This document describes four separate changes that all contribute to greatly reducing the shared memory detach time by providing granularity hints to the system. To realize the full benefit from these changes, you must set gh-chunks to a nonzero value, and the shared memory segments must be 8 MB aligned (both starting address and size aligned). Note that in Digital UNIX Version 4.0 or higher, the Segmented Shared Memory (SSM) functionality should be used instead of the granularity hints functionality in nearly all cases. Use the granularity hints technique only if you absolutely require the approximately 4% run-time performance gain from granularity hints. The manual tuning of both Digital UNIX and the database required for granularity hints makes it very complex and difficult to get right. You must rebuild your kernel, reboot, and manually adjust the kernel tuning/configuration parameters in /etc/sysconfigtab in order to enable use of granularity hints. Granularity Hints for shared memory segments -------------------------------------------- The granularity hints change makes use of the granularity hints bits in the page table entry (PTE) as specified in the Alpha Architecture. The granularity hints change is a Translation Buffer (TB) optimization that allows the TB to map more than a single page. Enabling the granularity hints change also enables the shared PTE change. The granularity hints change is enabled by setting the gh-chunks parameter in /etc/sysconfigtab. The gh-chunks parameter is the number of 4 Megabyte chunks reserved at boot time for shared memory use. An entry like the following must be in /etc/sysconfigtab: vm: gh-chunks=512 OR gh-chunks=0x200 Note: This reserves 2 Gigabytes for shared memory. The calculation is as follows: In decimal: 4194304 * 512 = 2147483648 In Hex: 0x400000 * 0x200 = 0x80000000 Please Note: Memory that is set aside at boot time for shared memory by specifying a value for gh-chunks will not be used for anything other than shared memory. Unused memory from this region is not given back to the system. The following short sequence of dbx commands that can be used to determine whether extra memory (beyond that required by the application) is being allocated at boot time with gh-chunks. # dbx -k /vmunix /dev/mem (dbx) px &gh_free_counts 0xfffffc0000681748 (dbx) 0xfffffc0000681748/4X fffffc0000681748: 0000000000000402 0000000000000004 fffffc0000681758: 0000000000000000 0000000000000002 (dbx) (dbx) q # The first number (402) is the number of 512 page chunks (4MB), the second number (4) is the 64 page chunks, the 0 is the 8 page chunks and the 2 is the number of 1 page chunks. Run this command while the application that allocates shared memory is running. The size of gh-chunks can be reduced such that there are only one or two 512-page chunks still free with the application running. Shared PTEs for shared memory segments - -------------------------------------- The shared PTEs change enables sharing of Level 3 page table entries in in the memory specified above by gh-chunks, resulting in a substantial savings of system memory. To be able to Level 3 PTEs, the shared memory segment attach address (in the shmat() system call) and the size of the shared memory segment (in shmget()) must be aligned on an 8 Megabyte boundry. This means that the lowest 23 bits of the address and size must be zero. Note 1: The attach address and the total size of the shared memory segment is specified by the application. Note 2: System V shared memory semantics allow a maximum shared memory segment size of (2 GB - 1 byte). Applications that need shared memory segments larger than 2 GB construct such regions by using multiple segments. The above notes imply that the total shared memory size specified by the user to the application must be 8 MB aligned. In addition, shm-max, the size specifying the maximum System V shared memory segment size, must be 8 MB aligned. If the total shared memory size specified to the application is greater than 2 GB, it is reasonable to set shm-max as follows: ipc: shm-max=2139095040 OR shm-max=0x7f800000 Note 3: This is the maximum value of shm-max that will allow sharing of PTEs. The maximum value for shm-max is calculated using the formula: shm-max = 2GB - 8MB The following dbx command can be used to determine if PTEs are being shared: dbx -k /vmunix /dev/mem (dbx) p *(vm_granhint_stats *)&gh_stats_store struct { total_mappers = 21 shared_mappers = 21 unshared_mappers = 0 total_unmappers = 21 shared_unmappers = 21 unshared_unmappers = 0 unaligned_mappers = 0 access_violations = 0 unaligned_size_requests = 0 unaligned_attachers = 0 wired_bypass = 0 wired_returns = 0 } (dbx) (dbx) q # The fields of interest are: shared_mappers unshared_mappers unaligned_attachers unaligned_size_requests For best performance you should see: shared_mappers = unshared_mappers = 0 unaligned_attachers = 0 unaligned_size_requests = 0 Because of the way shared memory is broken up into shared memory segments, there may be some unshared segments. This occurs when the starting address or the size is aligned on an 8 Megabyte boundary. This condition might be unavoidable in some cases. Please Note: In many cases the total_unmappers will be greater than the number of total_mappers. This is normal behavior. Rework of shared memory locking for better parallelism - ------------------------------------------------------ The shared memory locking rework changes a lock that was a single lock to a hashed array of locks. The size of the hashed array of locks is tunable with the vm-page-lock-count parameter in /etc/sysconfigtab: vm: vm-page-lock-count=64 Improvements in kernel malloc() garbage collection -------------------------------------------------- There are no tunable prameters relating to the improved kernel malloc() garbage collection. This is enabled by default and cannot be disabled. Other vm: /etc/sysconfigtab Options of Interest ----------------------------------------------- gh-min-seg-size=N specifies the size threshold where shared memory will start being allocated from the memory reserved by gh-chunks. The default is 8 Megabytes. gh-fail-if-no-mem=1 when set to 1, specifies for shmget() to return a failure if the requested size is larger than "gh-min-seg-size" and there is insufficient memory in the gh-chunks area to satisfy the request. If this flag is set to zero, the entire request will be satisfied from the pageable memory area if the request exceeds the gh-chunks area. The default is 1. There are messages that will print on the system console for unaligned size and attach address requests. The unaligned attach messages are throttled to one per shared memory segment.