6    Managing Memory Performance

You may be able to improve Tru64 UNIX performance by optimizing your memory resources. Usually, the best way to improve performance is to eliminate or reduce paging and swapping. This can be done by increasing memory resources.

This chapter describes:

• How the operating system allocates virtual memory to processes and to file system caches, and how memory is reclaimed (Section 6.1)

• How to configure swap space for high performance (Section 6.2)

• How to display information about memory usage (Section 6.3)

• The kernel subsystem attributes that you can modify to provide more memory resources to processes (Section 6.4)

• Paging and swapping operations (Section 6.5)

• How to reserve physical memory for shared memory (Section 6.6)

6.1    Virtual Memory Operation

The operating system allocates physical memory in 8-KB units called pages. The virtual memory subsystem tracks and manages all the physical pages in the system and efficiently distributes the pages among three areas:

• Static wired memory

  Allocated at boot time and used for operating system data and text and for system tables, static wired memory is also used by the metadata buffer cache, which holds recently accessed UNIX File System (UFS) and CD-ROM File System (CDFS) metadata.

  You can reduce the amount of static wired memory only by removing subsystems or by decreasing the size of the metadata buffer cache (see Section 6.1.2.1).

• Dynamically wired memory

  Dynamically wired memory is allocated at boot time and used for dynamically allocated data structures, such as system hash tables. User processes also allocate dynamically wired memory for address space by using virtual memory locking interfaces, including the mlock function. The amount of dynamically wired memory varies according to the demand. The vm subsystem attribute vm_syswiredpercent specifies the maximum amount of memory that a user process can wire (80 percent of physical memory, by default).

• Physical memory for processes and data caching

  Physical memory that is not wired is referred to as pageable memory. It is used for processes' most-recently accessed anonymous memory (modifiable virtual address space) and file-backed memory (memory that is used for program text or shared libraries). Pageable memory is also used to cache the most-recently accessed UFS file system data for reads and writes and for page faults from mapped file regions, in addition to AdvFS metadata and file data.

  The virtual memory subsystem allocates physical pages according to the process and file system demand. Because processes and file systems compete for a limited amount of physical memory, the virtual memory subsystem periodically reclaims the oldest pages by writing their contents to swap space or disk (paging). Under heavy loads, entire processes may be suspended to free large amounts of memory (swapping). You can control virtual memory operation by tuning various vm subsystem attributes, as described in this chapter.

You must understand memory operation to determine which tuning guidelines will improve performance for your workload. The following sections describe how the virtual memory subsystem:

• Tracks physical pages (Section 6.1.1)

• Allocates memory to file system buffer caches (Section 6.1.2)

• Allocates memory to processes (Section 6.1.3)

• Reclaims pages (Section 6.1.4)

6.1.1    Physical Page Tracking

The virtual memory subsystem tracks all the physical memory pages in the system. Page lists are used to identify the location and age of each page. The oldest pages are the first to be reclaimed. At any one time, each physical page can be found on one of the following lists:

• Wired list -- Pages that are wired and cannot be reclaimed

• Free list -- Pages that are clean and are not being used

  Page reclamation begins when the size of the free list decreases to a tunable limit.

• Active list -- Pages that are currently being used by processes or the UBC

  To determine which active pages should be reclaimed first, the page-stealer daemon identifies the oldest pages on the active list. The oldest pages that are being used by processes are designated inactive pages. The oldest pages that are being used by the UBC are designated UBC LRU (Unified Buffer Cache least-recently used) pages.

Use the vmstat command to determine the number of pages that are on the page lists. Remember that pages on the active list (the act field in the vmstat output) include both inactive and UBC LRU pages.

6.1.2    File System Buffer Cache Memory Allocation

The operating system uses caches to store file system user data and metadata. If the cached data is later reused, a disk I/O operation is avoided, which improves performance. This is because data can be retrieved from memory faster than a disk I/O operation.

The following sections describe these file system caches:

• Metadata buffer cache (Section 6.1.2.1)

• Unified Buffer Cache (Section 6.1.2.2)

6.1.2.1    Metadata Buffer Cache Memory Allocation

At boot time, the kernel allocates wired memory for the metadata buffer cache. The cache acts as a layer between the operating system and disk by storing recently accessed UFS and CDFS metadata, which includes file header information, superblocks, inodes, indirect blocks, directory blocks, and cylinder group summaries. Performance is improved if the data is later reused and a disk operation is avoided.

The metadata buffer cache uses bcopy routines to move data in and out of memory. Memory in the metadata buffer cache is not subject to page reclamation.

The size of the metadata buffer cache is specified by the value of the vfs subsystem bufcache attribute. See Section 9.1.3 for information on tuning the bufcache attribute.

6.1.2.2    Unified Buffer Cache Memory Allocation

The physical memory that is not wired is available to processes and to the Unified Buffer Cache (UBC), which compete for this memory.

The UBC functions as a layer between the operating system and disk by storing recently accessed file system data for reads and writes from conventional file activity and holding page faults from mapped file sections. UFS caches user and application data in the UBC. AdvFS caches user and application data and metadata in the UBC. File system performance is improved if the data and metadata is reused and in the UBC.

Figure 6-1 shows how the memory subsystem allocates physical memory to the UBC and for processes.

Figure 6-1:  UBC Memory Allocation

At any one time, the amount of memory allocated to the UBC and to processes depends on file system and process demands. For example, if file system activity is heavy and process demand is low, most of the pages will be allocated to the UBC, as shown in Figure 6-2.

Figure 6-2:  Memory Allocation During High File System Activity and No Paging Activity

In contrast, heavy process activity, such as large increases in the working sets for large executables, will cause the memory subsystem to reclaim UBC borrowed pages, down to the value of the ubc_borrowpercent attribute, as shown in Figure 6-3.

Figure 6-3:  Memory Allocation During Low File System Activity and High Paging Activity

The size of the UBC is specified by the value of the vfs subsystem UBC related attributes. See Section 9.1.2 for information on tuning the UBC related attribute.

6.1.3    Process Memory Allocation

Physical memory that is not wired is available to processes and the UBC, which compete for this memory. The virtual memory subsystem allocates memory resources to processes and to the UBC according to the demand, and reclaims the oldest pages if the demand depletes the number of available free pages.

The following sections describe how the virtual memory subsystem allocates memory to processes.

6.1.3.1    Process Virtual Address Space Allocation

The fork system call creates new processes. When you invoke a process, the fork system call:

1. Creates a UNIX process body, which includes a set of data structures that the kernel uses to track the process and a set of resource limitations. See fork(2) for more information.

2. Establishes a contiguous block of virtual address space for the process. Virtual address space is the array of virtual pages that the process can use to map into actual physical memory. Virtual address space is used for anonymous memory (memory that holds data elements and structures that are modified during process execution) and for file-backed memory (memory used for program text or shared libraries).

   Because physical memory is limited, a process' entire virtual address space cannot be in physical memory at one time. However, a process can execute when only a portion of its virtual address space (its working set) is mapped to physical memory. Pages of anonymous memory and file-backed memory are paged in only when needed. If the memory demand increases and pages must be reclaimed, the pages of anonymous memory are paged out and their contents moved to swap space, while the pages of file-backed memory are simply released.

3. Creates one or more threads of execution. The default is one thread for each process. Multiprocessing systems support multiple process threads.

Although the virtual memory subsystem allocates a large amount of virtual address space for each process, it uses only part of this space. Only 4 TB is allocated for user space. User space is generally private and maps to a nonshared physical page. An additional 4 TB of virtual address space is used for kernel space. Kernel space usually maps to shared physical pages. The remaining space is not used for any purpose.

Figure 6-4 shows the use of process virtual address space.

Figure 6-4:  Virtual Address Space Usage

6.1.3.2    Virtual Address to Physical Address Translation

When a virtual page is touched (accessed), the virtual memory subsystem must locate the physical page and then translate the virtual address into a physical address. Each process has a page table, which is an array containing an entry for each current virtual-to-physical address translation. Page table entries have a direct relation to virtual pages (that is, virtual address 1 corresponds to page table entry 1) and contain a pointer to the physical page and protection information.

Figure 6-5 shows the translation of a virtual address into a physical address.

Figure 6-5:  Virtual-to-Physical Address Translation

A process resident set is the complete set of all the virtual addresses that have been mapped to physical addresses (that is, all the pages that have been accessed during process execution). Resident set pages may be shared among multiple processes.

A process working set is the set of virtual addresses that are currently mapped to physical addresses. The working set is a subset of the resident set, and it represents a snapshot of the process resident set at one point in time.

6.1.3.3    Page Faults

When an anonymous (nonfile-backed) virtual address is requested, the virtual memory subsystem must locate the physical page and make it available to the process. This occurs at different speeds, depending on whether the page is in memory or on disk (see Figure 1-1).

If a requested address is currently being used (that is, the address is in the active page list), it will have an entry in the page table. In this case, the PAL code loads the physical address into the translation lookaside buffer, which then passes the address to the CPU. Because this is a memory operation, it occurs quickly.

If a requested address is not active in the page table, the PAL lookup code issues a page fault, which instructs the virtual memory subsystem to locate the page and make the virtual-to-physical address translation in the page table.

There are four different types of page faults:

• If a requested virtual address is being accessed for the first time, a zero-filled-on-demand page fault occurs. The virtual memory subsystem performs the following tasks:

  1. Allocates an available page of physical memory.

  2. Fills the page with zeros.

  3. Enters the virtual-to-physical address translation in the page table.

• If a requested virtual address has already been accessed and is located in the memory subsystem's internal data structures, a short page fault occurs. For example, if the physical address is located in the hash queue list or the page queue list, the virtual memory subsystem passes the address to the CPU and enters the virtual-to-physical address translation in the page table. This occurs quickly because it is a memory operation.

• If a requested virtual address has already been accessed, but the physical page has been reclaimed, the page contents will be found either on the free page list or in swap space. If a page is located on the free page list, it is removed from the hash queue and the free list and then reclaimed. This operation occurs quickly and does not require disk I/O.

  If a page is found in swap space, a page-in page fault occurs. The virtual memory subsystem copies the contents of the page from swap space into the physical address and enters the virtual-to-physical address translation in the page table. Because this requires a disk I/O operation, it requires more time than a memory operation.

• If a process needs to modify a read-only virtual page, a copy-on-write page fault occurs. The virtual memory subsystem allocates an available page of physical memory, copies the read-only page into the new page, and enters the translation in the page table.

The virtual memory subsystem uses the following techniques to improve process execution time and decrease the number of page faults:

• Mapping additional pages

  The virtual memory subsystem attempts to anticipate which pages the task will need next. Using an algorithm that checks which pages were most recently used, the number of available pages, and other factors, the subsystem maps additional pages along with the page that contains the requested address.

• Page coloring

  If possible, the virtual memory subsystem maps a process' entire resident set into the secondary cache and executes the entire task, text, and data within the cache.

  The vm subsystem attribute private_cache_percent specifies the percentage of the secondary cache that is reserved for anonymous memory. This attribute is used only for benchmarking. The default is to reserve 50 percent of the cache for anonymous memory and 50 percent for file-backed memory (shared). To cache more anonymous memory, increase the value of the private_cache_percent attribute.

6.1.4    Page Reclamation

Because memory resources are limited, the virtual memory subsystem must periodically reclaim pages. The free page list contains clean pages that are available to processes and the UBC. As the demand for memory increases, the list may become depleted. If the number of pages falls below a tunable limit, the virtual memory subsystem will replenish the free list by reclaiming the least-recently used pages from processes and the UBC.

To reclaim pages, the virtual memory subsystem:

1. Prewrites modified pages to swap space in an attempt to forestall a memory shortage. See Section 6.1.4.1 for more information.

2. Begins paging if the demand for memory is not satisfied, as follows:

   1. Reclaims pages that the UBC has borrowed and puts them on the free list.

   2. Reclaims the oldest inactive and UBC LRU pages from the active page list, moves the contents of the modified pages to swap space or disk, and puts the clean pages on the free list.

   3. If needed, more aggressively reclaims pages from the active list.

   See Section 6.1.4.2 for more information about reclaiming memory by paging.

3. Begins swapping if the demand for memory is not met. The virtual memory subsystem temporarily suspends processes and moves entire resident sets to swap space, which frees large numbers of pages. See Section 6.1.4.3 for information about swapping.

The point at which paging and swapping start and stop depends on the values of some vm subsystem attributes. Figure 6-6 shows some of the attributes that control paging and swapping.

Figure 6-6:  Paging and Swapping Attributes

Detailed descriptions of the attributes are as follows:

• vm_page_free_target -- Specifies a threshold value that stops paging. When the number of pages on the free page list reaches this value, paging stops. The default value of the vm_page_free_target attribute is based on the amount of memory in the system. Use Table 6-1 to determine the default value for your system.

  Table 6-1:  Default Values for vm_page_free_target Attribute

  Size of Memory	Value of vm_page_free_target
  Up to 512 MB	128
  513 MB to 1024 MB	256
  1025 MB to 2048 MB	512
  2049 MB to 4096 MB	768
  More than 4096 MB	1024

• vm_page_free_min -- Specifies a threshold value at which a page must be reclaimed for each page allocated. The default value is twice the value of the vm_page_free_reserved attribute.

• vm_page_free_reserved -- Specifies a threshold value that determines when memory is limited to privileged tasks. When the number of pages on the free page list falls below this value, only privileged tasks can get memory. The default value is 10 pages.

• vm_page_free_swap -- Specifies a threshold value that begins swapping of idle tasks. When the number of pages on the free page list falls below this value, idle task swapping begins. The default value is calculated using this formula:

  vm_page_free_min + ((vm_page_free_target - vm_page_free_min) / 2)
   
  

• vm_page_free_optimal -- Specifies a threshold value that begins hard swapping. When the number of pages on the free list falls below this value for five seconds, hard swapping begins. The first processes to be swapped out include those with the lowest scheduling priority and those with the largest resident set size. The default value is calculated using this formula:

  vm_page_free_min + ((vm_page_free_target - vm_page_free_min) / 2)
   
  

• vm_page_free_hardswap -- Specifies a threshold value that stops page swapping. When the number of pages on the free list reaches this value, paging stops. The default value is the value of the vm_page_free_target attribute multiplied by 16.

See Section 6.5 for information about modifying paging and swapping attributes.

The following sections describe the page reclamation procedure in detail.

6.1.4.1    Modified Page Prewriting

The virtual memory subsystem attempts to prevent memory shortages by prewriting modified inactive and UBC LRU pages to disk. To reclaim a page that has been prewritten, the virtual memory subsystem only needs to validate the page, which can improve performance. See Section 6.1.1 for information about page lists.

When the virtual memory subsystem anticipates that the pages on the free list will soon be depleted, it prewrites to disk the oldest modified (dirty) pages that are currently being used by processes or the UBC.

The value of the vm subsystem attribute vm_page_prewrite_target determines the number of inactive pages that the subsystem will prewrite and keep clean. The default value is vm_page_free_target * 2.

The vm_ubcdirtypercent attribute specifies the modified UBC LRU page threshold. When the number of modified UBC LRU pages is more than this value, the virtual memory subsystem prewrites to disk the oldest modified UBC LRU pages. The default value of the vm_ubcdirtypercent attribute is 10 percent of the total UBC LRU pages.

In addition, the sync function periodically flushes (writes to disk) system metadata and data from all unwritten memory buffers. For example, the data that is flushed includes, for UFS, modified inodes and delayed block I/O. Commands, such as the shutdown command, also issue their own sync functions. To minimize the impact of I/O spikes caused by the sync function, the value of the vm subsystem attribute ubc_maxdirtywrites specifies the maximum number of disk writes that the kernel can perform each second. The default value is five I/O operations per second.

6.1.4.2    Reclaiming Memory by Paging

When the memory demand is high and the number of pages on the free page list falls below the value of the vm subsystem attribute vm_page_free_target, the virtual memory subsystem uses paging to replenish the free page list. The page-out daemon and task swapper daemon are extensions of the page reclamation code, which controls paging and swapping.

The paging process is as follows:

1. The page reclamation code activates the page-stealer daemon, which first reclaims the clean pages that the UBC has borrowed from the virtual memory subsystem, until the size of the UBC reaches the borrowing threshold that is specified by the value of the ubc_borrowpercent attribute (the default is 20 percent). Freeing borrowed UBC pages is a fast way to reclaim pages, because UBC pages are usually not modified. If the reclaimed pages are dirty (modified), their contents must be written to disk before the pages can be moved to the free page list.

2. If freeing clean UBC borrowed memory does not sufficiently replenish the free list, a page out occurs. The page-stealer daemon reclaims the oldest inactive and UBC LRU pages from the active page list, moves the contents of the modified pages to disk, and puts the clean pages on the free list.

3. Paging becomes increasingly aggressive if the number of free pages continues to decrease. If the number of pages on the free page list falls below the value of the vm subsystem attribute vm_page_free_min (the default is 20 pages), a page must be reclaimed for each page taken from the list.

Figure 6-7 shows the movement of pages during paging operations.

Figure 6-7:  Paging Operation

Paging stops when the number of pages on the free list increases to the limit specified by the vm subsystem attribute vm_page_free_target. However, if paging individual pages does not sufficiently replenish the free list, swapping is used to free a large amount of memory (see Section 6.1.4.3).

6.1.4.3    Reclaiming Memory by Swapping

If there is a continuously high demand for memory, the virtual memory subsystem may be unable to replenish the free page list by reclaiming single pages. To dramatically increase the number of clean pages, the virtual memory subsystem uses swapping to suspend processes, which reduces the demand for physical memory.

The task swapper will swap out a process by suspending the process, writing its resident set to swap space, and moving the clean pages to the free page list. Swapping has a serious impact on system performance because a swapped out process cannot execute, and should be avoided on VLM systems and systems running large programs.

The point at which swapping begins and ends is controlled by a number of vm subsystem attributes, as follows:

• Idle task swapping begins when the number of pages on the free list falls below the value of the vm_page_free_swap attribute for a period of time. The task swapper suspends all tasks that have been idle for 30 seconds or more.

• Hard task swapping begins when the number of pages on the free page list falls below the value of the vm_page_free_optimal attribute for more than five seconds. The task swapper suspends, one at a time, the tasks with the lowest priority and the largest resident set size.

• Swapping stops when the number of pages on the free list increases to the value of the vm_page_free_hardswap attribute.

• A swap in occurs when the number of pages on the free list increases to the value of the vm_page_free_optimal attribute for a period of time. The value of the vm_inswappedmin attribute specifies the minimum amount of time, in seconds, that a task must remain in the inswapped state before it can be moved out of swap space. The default value is one second. The task's working set is then paged in from swap space, and the task can now execute. You can modify the value of the vm_inswappedmin attribute without rebooting the system.

You may be able to improve system performance by modifying the attributes that control when swapping begins and ends, as described in Section 6.5. Large-memory systems or systems running large programs should avoid paging and swapping, if possible.

Increasing the rate of swapping (swapping earlier during page reclamation) may increase throughput. As more processes are swapped out, fewer processes are actually executing and more work is done. Although increasing the rate of swapping moves long-sleeping threads out of memory and frees memory, it may degrade interactive response time because when an outswapped process is needed, it will have a long latency period.

Decreasing the rate of swapping (by swapping later during page reclamation) may improve interactive response time, but at the cost of throughput. See Section 6.5.2 for more information about changing the rate of swapping.

To facilitate the movement of data between memory and disk, the virtual memory subsystem uses synchronous and asynchronous swap buffers. The virtual memory subsystem uses these two types of buffers to immediately satisfy a page-in request without having to wait for the completion of a page-out request, which is a relatively slow process.

Synchronous swap buffers are used for page-in page faults and for swap outs. Asynchronous swap buffers are used for asynchronous page outs and for prewriting modified pages. See Section 6.5.7 for swap buffer tuning information.

6.2    Configuring Swap Space for High Performance

Use the swapon command to display swap space, and to configure additional swap space after system installation. To make this additional swap space permanent, use the vm subsystem attribute swapdevice to specify swap devices in the /etc/sysconfigtab file. For example:

vm:
     swapdevice=/dev/disk/dsk0b,/dev/disk/dsk0d

See Section 3.6 for information about modifying kernel subsystem attributes.

See Section 2.3.2.2 and Section 2.3.2.3 for information about swap space allocation modes and swap space requirements.

The following list describes how to configure swap space for high performance:

• Ensure that all your swap devices are configured when you boot the system, instead of adding swap space while the system is running.

• Use fast disks for swap space to decrease page-fault latency.

• Use disks that are not busy for swap space.

• Spread out swap space across multiple disks; do not put multiple swap partitions on the same disk. This makes paging and swapping more efficient and helps to prevent any single adapter, disk, or bus from becoming a bottleneck. The page reclamation code uses a form of disk striping (known as swap-space interleaving) that improves performance when data is written to multiple disks.

• Spread out your swap disks across multiple I/O buses to prevent a single bus from becoming a bottleneck.

• Configure multiple swap devices as individual devices (or LSM volumes) instead of striping the devices and configuring only one logical swap device.

• If you are paging heavily and cannot increase the amount of memory in your system, consider using RAID 5 for swap devices. See Chapter 8 for more information about RAID 5.

See the System Administration manual for more information about adding swap devices. See Chapter 8 for more information about configuring and tuning disks for high performance and availability.

6.3    Displaying Memory Information

Table 6-2 describes the tools that you can use to display memory usage information.

Table 6-2:  Tools to Display Virtual Memory and UBC

The following sections describe some of these tools in detail.

6.3.1    Displaying Memory by Using the vmstat Command

To display the virtual memory, process, and CPU statistics, enter:

# /usr/ucb/vmstat

For example, to display statistics

Information similar to the following is displayed:

Virtual Memory Statistics: (pagesize = 8192)
procs        memory            pages                       intr        cpu
r  w  u  act  free wire  fault cow zero react pin pout   in  sy  cs  us sy  id
2 66 25  6417 3497 1570  155K  38K  50K    0  46K    0    4 290 165   0  2  98
4 65 24  6421 3493 1570   120    9   81    0    8    0  585 865 335  37 16  48
2 66 25  6421 3493 1570    69    0   69    0    0    0  570 968 368   8 22  69
4 65 24  6421 3493 1570    69    0   69    0    0    0  554 768 370   2 14  84
4 65 24  6421 3493 1570    69    0   69    0    0    0  865  1K 404   4 20  76
  [1]           [2]            [3]                        [4]         [5]
 
 

The first line of the vmstat output is for all time since a reboot, and each subsequent report is for the last interval.

The vmstat command includes information that you can use to diagnose CPU and virtual memory problems. Examine the following fields:

1. Process information (procs):

   • r -- Number of threads that are running or can run.

   • w -- Number of threads that are waiting interruptibly (waiting for an event or a resource, but can be interrupted or suspended). For example, the thread can accept user signals or be swapped out of memory.

   • u -- Number of threads that are waiting uninterruptibly (waiting for an event or a resource, but cannot be interrupted or suspended). For example, the thread cannot accept user signals; it must come out of the wait state to take a signal. Processes that are waiting uninterruptibly cannot be stopped by the kill command.

   [Return to example]

2. Virtual memory information (memory):

   • act -- Number of pages on the active list, including inactive pages and UBC LRU pages.

   • free -- Number of pages on the free list.

   • wire -- Number of pages on the wired list. Pages on the wired list cannot be reclaimed.

   See Section 6.1.1 for more information on page lists. [Return to example]

3. Paging information (pages):

   • fault -- Number of address translation faults.

   • cow -- Number of copy-on-write page faults. These page faults occur if the requested page is shared by a parent process and a child process, and one of these processes needs to modify the page. If a copy-on-write page fault occurs, the virtual memory subsystem loads a new address into the translation buffer, and then copies the contents of the requested page into this address, so that the process can modify it.

   • zero -- Number of zero-filled-on-demand page faults. These page faults occur if a requested page is not located in the internal data structures and has never been referenced. If a zero-filled-on-demand page fault occurs, the virtual memory subsystem allocates an available page of physical memory, fills the page with zeros, and then enters the address into the page table.

   • react -- Number of pages that have been faulted (touched) while on the inactive page list.

   • pin -- Number of requests for pages from the page-stealer daemon.

   • pout -- Number of pages that have been paged out to disk.

   [Return to example]

4. Interrupt information (intr):

   • in -- Number of nonclock device interrupts per second.

   • sy -- Number of system calls called per second.

   • cs -- Number of task and thread context switches per second.

   [Return to example]

5. CPU usage information (cpu):

   • us -- Percentage of user time for normal and priority processes. User time includes the time the CPU spent executing library routines.

   • sy -- Percentage of system time. System time includes the time the CPU spent executing system calls.

   • id -- Percentage of idle time.

   See Section 7.1.2 for information about using the vmstat command to monitor CPU usage. [Return to example]

To use the vmstat command to diagnose a memory performance problem:

• Check the size of the free page list (free). Compare the number of free pages to the values for the active pages (act) and the wired pages (wire). The sum of the free, active, and wired pages should be close to the amount of physical memory in your system. Although the value for free should be small, if the value is consistently small (less than 128 pages) and accompanied by excessive paging and swapping, you may not have enough physical memory for your workload.

• Examine the pout field. If the number of page outs is consistently high, you may have insufficient memory.

• The following command output may indicate that the size of the UBC is too small for your configuration:

  • The output of the vmstat or monitor command shows excessive file system page-in activity, but little or no page-out activity or shows a very low free page count.

  • The output of the iostat command shows little or no swap disk I/O activity or shows excessive file system I/O activity. See Section 8.2 for more information.

Excessive paging also can increase the miss rate for the secondary cache, and may be indicated by the following output:

• The output of the ps command shows high task swapping activity. See Section 6.3.2 for more information.

• The output of the swapon command shows excessive use of swap space. See Section 6.3.3 for more information.

To display statistics about physical memory use, enter:

# vmstat -P

Information similar to the following is displayed:

Total Physical Memory =   512.00 M
                      =    65536 pages
Physical Memory Clusters:
 
 start_pfn     end_pfn        type  size_pages / size_bytes
         0         256         pal         256 /    2.00M
       256       65527          os       65271 /  509.93M
     65527       65536         pal           9 /   72.00k
 
Physical Memory Use:
 
 start_pfn     end_pfn        type  size_pages / size_bytes
       256         280   unixtable          24 /  192.00k
       280         287    scavenge           7 /   56.00k
       287         918        text         631 /    4.93M
       918        1046        data         128 /    1.00M
      1046        1209         bss         163 /    1.27M
      1210        1384      kdebug         174 /    1.36M
      1384        1390     cfgmgmt           6 /   48.00k
      1390        1392       locks           2 /   16.00k
      1392        1949   unixtable         557 /    4.35M
      1949        1962        pmap          13 /  104.00k
      1962        2972    vmtables        1010 /    7.89M
      2972       65527     managed       62555 /  488.71M
                             ============================
         Total Physical Memory Use:      65270 /  509.92M
 
Managed Pages Break Down:
 
       free pages = 1207
     active pages = 25817
   inactive pages = 20103
      wired pages = 15434
        ubc pages = 15992
        ==================
            Total = 78553
 
WIRED Pages Break Down:
 
   vm wired pages = 1448
  ubc wired pages = 4550
  meta data pages = 1958
     malloc pages = 5469
     contig pages = 159
    user ptepages = 1774
  kernel ptepages = 67
    free ptepages = 9
        ==================
            Total = 15434

See Section 6.4 for information about increasing memory resources.

6.3.2    Displaying Memory by Using the ps Command

To display the current state of the system processes and how they use memory, enter:

# /usr/ucb/ps aux

Information similar to the following is displayed:

USER  PID  %CPU %MEM   VSZ   RSS  TTY S    STARTED      TIME  COMMAND
chen  2225  5.0  0.3  1.35M  256K p9  U    13:24:58  0:00.36  cp /vmunix /tmp
root  2236  3.0  0.5  1.59M  456K p9  R  + 13:33:21  0:00.08  ps aux
sorn  2226  1.0  0.6  2.75M  552K p9  S  + 13:25:01  0:00.05  vi met.ps
root   347  1.0  4.0  9.58M  3.72 ??  S      Nov 07 01:26:44  /usr/bin/X11/X -a
root  1905  1.0  1.1  6.10M  1.01 ??  R    16:55:16  0:24.79  /usr/bin/X11/dxpa
mat   2228  0.0  0.5  1.82M  504K p5  S  + 13:25:03  0:00.02  more
mat   2202  0.0  0.5  2.03M  456K p5  S    13:14:14  0:00.23  -csh (csh)
root     0  0.0 12.7   356M  11.9 ??  R <  Nov 07 3-17:26:13  [kernel idle]
             [1] [2]   [3] [4]      [5]               [6]     [7]
 
 

The ps command displays a snapshot of system processes in order of decreasing CPU usage, including the execution of the ps command itself. By the time the ps command executes, the state of system processes has probably changed.

The ps command output includes the following information that you can use to diagnose CPU and virtual memory problems:

1. Percentage of CPU time usage (%CPU). [Return to example]

2. Percentage of real memory usage (%MEM). [Return to example]

3. Process virtual address size (VSZ)--This is the total amount of anonymous memory allocated to the process (in bytes). [Return to example]

4. Real memory (resident set) size of the process (RSS)--This is the total amount of physical memory (in bytes) mapped to virtual pages (that is, the total amount of memory that the application has physically used). Shared memory is included in the resident set size figures; as a result, the total of these figures may exceed the total amount of physical memory available on the system. [Return to example]

5. Process status or state (S) -- This specifies whether a process is in one of the following states:

   • Runnable (R)

   • Sleeping (S) -- Process has been waiting for an event or a resource for less than 20 seconds, but it can be interrupted or suspended. For example, the process can accept user signals or be swapped out.

   • Uninterruptible sleeping (U) -- Process is waiting for an event or a resource, but cannot be interrupted or suspended. You cannot use the kill command to stop these processes; they must come out of the wait state to accept the signal.

   • Idle (I) -- Process has been sleeping for more than 20 seconds.

   • Stopped (T) -- Process has been stopped.

   • Halted (H) -- Process has been halted.

   • Swapped out (W) -- Process has been swapped out of memory.

   • Locked into memory (L) -- Process has been locked into memory and cannot be swapped out.

   • Has exceeded the soft limit on memory requirements (>)

   • A process group leader with a controlling terminal (+)

   • Has a reduced priority (N)

   • Has a raised priority (<)

   [Return to example]

6. Current CPU time used (TIME), in the format hh:mm:ss.ms. [Return to example]

7. The command that is running (COMMAND). [Return to example]

From the output of the ps command, you can determine which processes are consuming most of your system's CPU time and memory resources, and whether processes are swapped out. Concentrate on processes that are running or paging. Here are some concerns to keep in mind:

• If a process is using a large amount of memory (see the RSS and VSZ fields), the process may have excessive memory requirements. See Section 11.2 for information about decreasing an application's use of memory.

• If duplicate processes are running, use the kill command to terminate duplicate processes. See kill(1) for more information.

• If a process is using a large amount of CPU time, it may be in an infinite loop. You may have to use the kill command to terminate the process and then correct the problem by making changes to its source code.

  You can also use the Class Scheduler to allocate a percentage of CPU time to a specific task or application (see Section 7.2.2) or lower the process' priority by using either the nice or renice command. These commands have no effect on memory usage by a process. See nice(8) or renice(8) for more information.

• Check the processes that are swapped out. Examine the S (state) field. A W entry indicates a process that has been swapped out. If processes are continually being swapped out, this could indicate a lack of memory resources. See Section 6.4 for information about increasing memory resources.

6.3.3    Displaying Swap Space Usage by Using the swapon Command

To display information about your swap device configuration, including the total amount of allocated swap space, the amount of swap space that is being used, and the amount of free swap space, enter:

# /usr/sbin/swapon -s

Infomation for each swap partition is displayed similar to the following:

Swap partition /dev/disk/dsk1b (default swap):     
    Allocated space:        16384 pages (128MB)     
    In-use space:           10452 pages ( 63%)     
    Free space:              5932 pages ( 36%)  
 
Swap partition /dev/disk/dsk4c:
    Allocated space:        128178 pages (1001MB)     
    In-use space:            10242 pages (  7%)     
    Free space:             117936 pages ( 92%)   
 
Total swap allocation:     
 
    Allocated space:        144562 pages (1.10GB)     
    Reserved space:          34253 pages ( 23%)     
    In-use space:            20694 pages ( 14%)     
    Available space:        110309 pages ( 76%)

You can configure swap space when you first install the operating system, or you can add swap space at a later date. Application messages, such as the following, usually indicate that not enough swap space is configured into the system or that a process limit has been reached:

struct {
    read_looks = 2059022
    read_hits = 2022488
    read_miss = 36506
    alloc_error = 0
    alloc_in_cache = 0
}
(dbx)

# /usr/ucb/dbx -k /vmunix /dev/mem
 
(dbx) px &gh_free_counts
0xfffffc0000681748
(dbx) 0xfffffc0000681748/4X
fffffc0000681748:  0000000000000402 0000000000000004
fffffc0000681758:  0000000000000000 0000000000000002
(dbx)

# /usr/ucb/dbx -k /vmunix /dev/mem
 
(dbx) p *(vm_granhint_stats *)&gh_stats_store
	struct {
	    total_mappers = 21
	    shared_mappers = 21
	    unshared_mappers = 0
	    total_unmappers = 21
	    shared_unmappers = 21
	    unshared_unmappers = 0
	    unaligned_mappers = 0
	    access_violations = 0
	    unaligned_size_requests = 0
	    unaligned_attachers = 0
	    wired_bypass = 0
	    wired_returns = 0
	} 
	(dbx)