5    Kernel, Symmetric Multiprocessing, Virtual Memory, and Device Support

This chapter discusses the Tru64 UNIX kernel, symmetric multiprocessing, the virtual memory subsystem, and support for hardware devices. It is organized as follows:

• The Tru64 UNIX kernel (Section 5.1)

• Symmetric multiprocessing (Section 5.2)

• The virtual memory subsystem (Section 5.3)

• Tru64 UNIX support for various devices (Section 5.4)

5.1    The Tru64 UNIX Kernel

The kernel manages the Tru64 UNIX system resources. It can be adjusted for maximum performance by setting system attributes. Furthermore kernel tuning and debugging tools allow the examination of these attributes.

5.1.1    Kernel Tuning

Tru64 UNIX includes various subsystems that are used to define or extend the kernel. Kernel variables control subsystem behavior or track subsystem statistics after boot time.

Kernel variables are assigned default values at boot time. For certain configurations and workloads, especially memory or network-intensive systems, the default values of some attributes may not be appropriate, so you must modify these values to provide optimal performance.

Although you can use the dbx debugger to directly change variable values on a running kernel, Compaq recommends that you use kernel subsystem attributes to access the kernel variables.

Subsystem attributes are managed by the configuration manager server, cfgmgr. You can display and modify attributes by using sysconfig and sysconfigdb commands and by using the Kernel Tuner, dxkerneltuner. In some cases, you can use the sysconfig to modify attributes while the system is running. However, these run-time modifications are lost when the system reboots.

For more information, see the System Configuration and Tuning guide.

5.1.2    System Attributes and Parameters

The following system attributes have been added to support the Advanced File System:

• AdvfsMinFreeAccess

• AdvfsMaxFreeAccessPercent

• AdvfsDomainPanicLevel

• AdvfsCacheHashSize

The AdvfsMinFreeAccess and AdvfsMaxFreeAccessPercent attributes provide you with more control over the number of available AdvFS access structures on your system. Access structures are the in-memory representation of an AdvFS file. These attributes supersede the AdvfsAccessCleanupPercent attribute, which has been removed.

The AdvfsDomainPanicLevel attribute controls what happens during an AdvFS domain panic. By default, when an AdvFS domain panic occurs, this attribute enables the system to create a crash dump without crashing the system.

The AdvfsCacheHashSize attribute allows you to adjust the size of AdvFS buffer cache hash table size. AdvFS configures its buffer cache hash table size based on the AdvfsCacheHashSize value.

Additionally, the default value of the AdvfsAccessMaxPercent attribute has changed.

For more information, see the AdvFS Administration and System Configuration and Tuning guides and the sys_attrs_advfs(5) reference page.

5.1.3    Enhanced Kernel Debugging

The dbx debugger supports a read-only examination of a locally running kernel and the debugging of kernel core files through the use of the -k option.

The following two features have been added to dbx:

• A -remote option to enable the remote, breakpoint debugging of a running kernel across a serial line.

  The protocol is multibyte; it supports caching multithread extension.

• A front-end to dbx called kdbx, which supports the entire suite of dbx commands, as well as a C library API that allows programmers to write C programs to extract and format kernel data more easily than they can with dbx -kor dbx -remote.

  The kdbx front-end ships with several ready-made extensions in the file /usr/var/kdbx.

For more information on kernel debugging, see the Kernel Debugging guide.

5.2    Symmetric Multiprocessing

Symmetric multiprocessing (SMP) is the ability of two or more processes (or multiple threads of a threaded application) to execute simultaneously on two or more CPUs. This concurrency of execution greatly improves performance. Additionally, it provides the opportunity to extend the life and increase the cost-effectiveness of multiprocessor systems by adding CPU cards (and their compute power) to multiprocessors rather than buying more systems.

Tru64 UNIX supports an implementation of SMP that is designed to optimize the performance of compute servers (systems dedicated to compute-bound, multithreaded applications) and data servers (file servers, DBMS servers, TP systems, and mail routers that serve a large number of network clients). In addition, Tru64 UNIX supports multithreaded application development in an SMP environment. Note that SMP does not adversely affect using a multiprocessor as a timesharing system.

Tru64 UNIX SMP uses the following:

• Simple locks (also called spin locks, because they "spin" for a specified period of time waiting for held locks to be freed before timing out).

• Complex locks (read/write locks that can block waiting for a lock to be freed).

• Funneling, which is used in rare cases where locks would not be of benefit. Funneling forces a proces to execute on a specified CPU.

Tru64 UNIX SMP achieves as much concurrency as possible by reducing the size of the system state that must be protected by locks, thereby reducing the necessity for locks and their overhead.

Tru64 UNIX, including its kernel, is fully parallel so that multiple processes or multiple threads can run simultaneously on multiple CPUs. The operating system uses concurrency and its locking strategy to ensure the integrity of the same kernel data structures. Multiple processes and multiple threads can access the same kernel data structures, but Tru64 UNIX makes sure that this access is performed in a logical order. Multiple processes and multiple threads cannot hold and request each others' locks, deadlocking the system. There are no architectural limits on the number of CPUs supported.

Tru64 UNIX SMP also supports processor affinity, the ability to bind a particular process to a specified CPU, and load balancing, whereby the scheduler attempts to distribute all runable processes across all available CPUs. (Note that load balancing will not override processor affinity.)

To improve performance, the scheduler also attempts to execute each process on the last CPU where it ran to take advantage of any state that may be left in that CPU's cache.

SMP is configurable and any of the following five modes can be configured at system boot time:

• Uniprocessing

• Optimized real-time preemption

• Optimized SMP

• Optimized real-time preemption and SMP

• Lock debug mode

When uniprocessing is set, only those locks required to support multiple threads are compiled into the kernel at system boot time.

When lock debug mode is set, the system:

• Checks the lock hierarchy and minimum system priority level (SPL)

• Stores debugging information by classes and maintains lock statistics

• Records the simple locks that are held by each CPU in CPU-specific arrays

• Records all of the complex locks that a thread is holding in the thread structure

You can use the dbx debugger to access this debugging information.

In addition, the development environment supports multithreaded application development. The dbx, profile, and pixie utilities support multiple threads, and the system includes thread-safe libraries.

For information on the Tru64 UNIX development environment and the threads package that Tru64 UNIX supports, see the Programmer's Guide and the Guide to the POSIX Threads Library. For information on configuring SMP, see the System Administration and System Configuration and Tuning guides.

5.3    Virtual Memory

The virtual memory subsystem performs the following functions:

• Allocates memory to processes

• Tracks and manages all the pages in the system

• Uses paging and swapping to ensure that there is enough memory for processes to run and to cache file system I/O

The total amount of physical memory is determined by the capacity of the memory boards installed in your system. The system distributes this memory in 8 KB units called pages. The system distributes pages of physical memory among three areas:

• Wired memory

  Memory is wired statically at boot time and dynamically at run time.

  At boot time, the operating system and the Privileged Architecture Library (PAL) code wire a contiguous portion of physical memory to perform basic system operations. Static wired memory is reserved for operating system data and text, system tables, the metadata buffer cache, which temporarily holds recently accessed UNIX File System (UFS) and CD-ROM File System (CDFS) metadata, and the Advanced File System (AdvFS) buffer cache. Static wired memory cannot be reclaimed through paging.

• Virtual memory

  The virtual memory subsystem uses a portion of physical memory to cache processes' most recently accessed anonymous memory (modifiable virtual address space) and file-backed memory. The subsystem allocates memory to competing processes and tracks the distribution of all the physical pages. This memory can be reclaimed through paging and swapping.

• Unified Buffer Cache

  The Unified Buffer Cache (UBC) uses a portion of physical memory to cache most recently accessed file system data. The UBC contains actual file data for reads and writes and for page faults from mapped file regions and also AdvFS metadata. By functioning as a layer between the operating system and the storage subsystem, the UBC can decrease the number of disk operations. This memory can be reclaimed through paging.

The virtual memory subsystem and the UBC compete for the physical pages that are not wired. Pages are allocated to processes and to the UBC, as needed. When the demand for memory increases, the oldest (least recently used) pages are reclaimed from the virtual memory subsystem and the UBC, are moved to swap space, and are then reused. Various attributes control the amount of memory available to the virtual memory subsystem and the UBC and the rate of page reclamation.

5.3.1    Managing and Tracking Pages

The virtual memory subsystem allocates physical pages to processes and the UBC, as needed. Because physical memory is limited, these pages must be periodically reclaimed so that they can be reused.

The virtual memory subsystem uses page lists to track the location and age of all the physical memory pages. At any one time, each physical page can be found on one of the following lists:

• Wired list

  Pages that are wired and cannot be reclaimed.

• Free list

  Pages that are clean and are not being used. The size of this list controls when page reclamation occurs.

• Active list

  Pages that are being used by the virtual memory subsystem or the UBC.

  To determine which pages should be reclaimed first, the page stealer daemon identifies the oldest pages on the active list and designates the least recently used pages as follows:

  • Inactive pages are the oldest pages that are being used by the virtual memory subsystem.

  • UBC least recently used pages are the oldest pages that are being used by the UBC.

5.3.2    Prewriting Modified Pages

The virtual memory subsystem attempts to keep memory pages clean to ease the recovery of memory shortages. When the virtual memory subsystem anticipates that the pages on the free list will soon be depleted, it prewrites to swap space the oldest modified (dirty) inactive pages. In addition, when the number of modified UBC least recently used pages exceeds 10 percent of the total UBC least recently used pages, the virtual memory subsystem prewrites to swap space the oldest modified UBC least recently used pages.

5.3.3    Using Attributes to Control Paging and Swapping

When the demand for memory significantly depletes the free list, paging begins. The virtual memory subsystem takes the oldest inactive pages and UBC least recently used pages, moves the contents of the modified pages to swap space, and puts the clean pages on the free list, where they can be reused.

If the free page list cannot be replenished by reclaiming individual pages, swapping begins. Swapping temporarily suspends processes and moves entire resident sets to swap space, which frees large amounts of physical memory.

The point at which paging and swapping start and stop depends on the values of various tunable virtual memory subsystem kernel attributes.

Because the UBC competes with the virtual memory subsystem for the physical pages that are not wired by the kernel, the allocation of memory to the UBC can affect file system performance and paging and swapping activity. The UBC is dynamic and consumes varying amounts of memory while responding to changing file system demands.

By default, the UBC can consume up to 100 percent of memory. However, part of the memory allocated to the UBC is only borrowed from the virtual memory subsystem. When paging starts, borrowed UBC pages are the first to be reclaimed. The amount of memory allocated to the UBC can be controlled by various virtual memory subsystem kernel attributes.

5.3.4    Paging Operation

When the memory demand is high and the number of pages on the free page list falls below the paging threshold, the virtual memory subsystem uses paging to replenish the free page list. The page reclamation code controls paging and swapping. The page out daemon and task swapper daemon are extensions of the page reclamation code.

The page reclamation code activates the page stealer daemon, which first reclaims the pages that the UBC has borrowed from the virtual memory subsystem, until the size of the UBC reaches the borrowing threshold. (The default is 20 percent.) If the reclaimed pages are dirty (modified), their contents must be written to disk before the pages can be moved to the free page list. Freeing borrowed UBC pages is a fast way to reclaim pages, because UBC pages are usually unmodified.

If freeing UBC borrowed memory does not sufficiently replenish the free list, a page out occurs. The page stealer daemon reclaims the oldest inactive pages and UBC least recently used pages.

Paging becomes increasingly aggressive if the number of free pages continues to decrease. If the number of pages on the free page list falls below 20 pages (the default), a page must be reclaimed for each page taken from the list. To prevent deadlocks, when the number of pages on the free page list falls below 10 pages (the default), only privileged tasks can get memory until the free page list is replenished. Both these limits are controlled by tunable attributes.

Page out stops when the number of pages on the free list rises above the paging threshold. If paging individual pages does not sufficiently replenish the free list, swapping is used to free a large amount of memory.

5.3.5    Swapping Operation

If there is a high demand for memory, the virtual memory subsystem may be unable to replenish the free list by reclaiming pages. Swapping reduces the demand for physical memory by suspending processes, which dramatically increases the number of pages on the free list. To swap out a process, the task swapper suspends the process, writes its resident set to swap space, and moves the clean pages to the free list. Swapping can have a serious impact on system performance.

Idle task swapping begins when the number of pages on the free list falls below the swapping threshold (the default is 74 pages) for a period of time. The task swapper then suspends all tasks that have been idle for 30 seconds or more.

If the number of pages on the free list continues to decrease, hard swapping begins. The task swapper suspends, one at a time, the tasks with the lowest priority and the largest resident set size.

Swapping of an individual task stops when the number of pages on the free list reaches the high water swapping threshold. (The default is 1280.)

A swap in occurs when the number of pages on the free list has been sufficiently replenished for a period of time. The task's working set is paged in from swap space and it can now execute. By default, a task must remain in the swapped-in state for one second before it can be swapped out.

Increasing the rate of swapping (swapping earlier during page reclamation) increases throughput. As more processes are swapped out, fewer processes are actually executing and more work is done. Although increasing the rate of swapping moves long-sleeping threads out of memory and frees memory, it degrades interactive response time. Swapped out processes have a long latency.

Decreasing the rate of swapping (swapping later during page reclamation), improves interactive response time, but at the cost of throughput.

5.3.6    Swap Space Allocation Mode

You can use two modes to allocate swap space. The modes differ in how the virtual memory subsystem reserves swap space for anonymous memory (modifiable virtual address space). Anonymous memory is memory that is not backed by a file, but is backed by swap space (for example, stack space, heap space, and memory allocated by the malloc function). Neither mode has a performance benefit attached to it:

• Immediate mode

  This mode reserves swap space when a process first allocates anonymous memory. Immediate mode is the default swap space allocation mode and is also called eager mode.

  This mode may cause the system to reserve an unnecessarily large amount of swap space for processes. However, it ensures that swap space will be available to processes if it is needed.

• Deferred mode

  This mode reserves swap space only if the virtual memory subsystem needs to write a modified virtual page to swap space. It postpones the reservation of swap space for anonymous memory until it is actually needed. Deferred mode is also called lazy mode.

  This mode requires less swap space than immediate mode and may cause the system to run faster, because it requires less swap space bookkeeping. However, because deferred mode does not reserve swap space in advance, the swap space may not be available when a process needs it, and processes may be killed asynchronously.

In addition, you can override the system-wide swap space allocation mode for a specific command or application by using the swapon command. For more information, see the swapon(8) reference page.

5.3.7    Using Swap Buffers

To facilitate the movement of data between memory and disk, the virtual memory subsystem uses synchronous and asynchronous swap buffers. The virtual memory subsystem uses these two types of buffers to immediately satisfy a page in request without having to wait for the completion of a page out request, which is a relatively slow process.

Synchronous swap buffers are used for page-in page faults and for swap outs. Asynchronous swap buffers are used for asynchronous page outs and for prewriting modified pages.

5.3.8    Unified Buffer Cache

The Unified Buffer Cache (UBC) is a Tru64 UNIX virtual memory feature. The UBC uses a portion of the machine's physical memory to cache the most recently accessed file system data. The UBC contains actual file data, which includes reads and writes from conventional file activity, page faults from mapped file sections, and AdvFS metadata.

The UBC shares (contends for) physical memory pages with the virtual memory subsystem, but not pages that are wired by the kernel. The UBC is dynamic, consuming varying amounts of memory in response to changes in file system demands for its service. For information about the UBC, see the System Configuration and Tuning guide.

5.4    Device Support

The kernel supports hot-swap I/O devices, which provides the capability to automatically fault-in a device driver when a hot-swappable I/O device comes on line.

When the hardware code detects a new device and determines that the device driver is not present in the kernel, it can make a kernel function call to automatically load the device's driver into the kernel. Additionally, hot-swapping provides the flexibility of not having to prebuild the kernel subsystem or driver into the kernel. Instead, it can be faulted in when the device is first accessed.