5    Tuning System Resource Allocation

The Tru64 UNIX operating system sets resource limits at boot time. These limits control the size of system tables, virtual address space, and other system resources.

The default system resource limits are appropriate for most configurations. However, if your system has a large amount of memory, is running a program that requires extensive resources, or running a large-memory application, you may need to increase the system limits by modifying subsystem attributes.

This chapter describes how to increase the following system-wide limits:

• Process limits (Section 5.1)

• Program size limits (Section 5.2)

• Address space limits (Section 5.3)

• Interprocess communication (IPC) limits (Section 5.4)

• Open file limits (Section 5.5)

Instead of modifying system-wide limits, you can use the setrlimit function to control the consumption of system resources by a specific process and its child processes. See setrlimit(2) for information.

5.1    Tuning Process Limits

Tru64 UNIX uses process limits that are appropriate for most configurations. However, if your applications are memory-intensive or you have a very-large memory (VLM) system or an Internet server (including, Web, proxy, firewall, or gateway servers), you may want to increase the process limits. Because increasing process limits increases the amount of wired memory in the system, increase the limits only if your system requires more resources.

The following sections describe how to increase these limits:

• System tables and data structures (Section 5.1.1)

• Maximum number of processes (Section 5.1.2)

• Maximum number of threads (Section 5.1.3)

5.1.1    Increasing System Tables and Data Structures

System algorithms use the proc subsystem attribute maxusers to size various system data structures and system tables, such as the system process table, which determines how many active processes can be running at one time.

The value of the maxusers attribute is used to set the default values for some subsystem attributes that set system limits, including the max_proc_per_user, max_threads_per_user, min_free_vnodes, and name_cache_size attributes.

Performance Benefit and Tradeoff

Increasing the value of maxusers provides more system resources to processes. However, increasing the resources available to users will increase the amount of wired memory.

You can modify the maxusers attribute without rebooting the system.

When to Tune

If you have a large-memory system or Internet server, or your system experiences a lack of resources, increase the value of the maxusers attribute. A lack of resources can be indicated by a No more processes, Out of processes, or pid table is full message.

Recommended Values

The default value assigned to the maxusers attribute depends on the amount of memory in the system. Table 5-1 shows the default value of the maxusers attribute for systems with various amounts of memory.

Table 5-1:  Default Values for the maxusers Attribute

To determine an appropriate value for the maxusers attribute, double the default value until you notice a performance improvement. If you have an Internet server, you can increase the value of the maxusers attribute to 2048. It is recommended that you not increase the value of the maxusers attribute to more than 2048.

If you increase the value of maxusers, you may want to increase the value of the max_vnodes attribute proportionally (see Section 5.5.1).

You must not decrease the default value of the maxusers attribute.

See Section 3.6 for information about modifying kernel subsystem attributes.

5.1.2    Increasing the Maximum Number of Processes

The proc subsystem attribute max_proc_per_user specifies the maximum number of processes that can be allocated at any one time to each user, except superuser.

Performance Benefit and Tradeoff

Increasing the value of max_proc_per_user provides more system resources to processes.

When to Tune

If your system experiences a lack of processes or you have a very-large memory (VLM) system or an Internet server, you may want to increase the value of the max_proc_per_user attribute.

You cannot modify the max_proc_per_user attribute without rebooting the system.

Recommended Values

The default value of the max_proc_per_user attribute is based on the maxusers attribute. If you want to increase the maximum number of processes, you can increase the value of the maxusers attribute (Section 5.1.1). As an alternative, you can specify a value for the max_proc_per_user attribute that is equal to or greater than the maximum number of processes that will be running on the system at one time. If you have a Web server, these processes include CGI processes.

If you have an Internet server, increase the value of the max_proc_per_user attribute to 512.

If you specify a value of 0 (zero) for the max_proc_per_user attribute, there is no limit on processes.

See Section 3.6 for information about modifying kernel subsystem attributes.

5.1.3    Increasing the Maximum Number of Threads

The proc subsystem attribute max_threads_per_user specifies the maximum number of threads that can be allocated at any one time to each user, except superuser.

Performance Benefit and Tradeoff

Increasing the value of max_threads_per_user provides more system resources to processes.

You cannot modify the max_proc_per_user attribute without rebooting the system.

When to Tune

If your system experiences a lack of threads or you have a VLM system or an Internet server, you may want to increase the value of the max_threads_per_user attribute.

Recommended Values

The default value of the max_threads_per_user attribute is based on the value of the maxusers attribute. If you want to increase the maximum number of threads, you can modify the maxusers attribute (Section 5.1.1). As an alternative, you can specify a value for the max_threads_per_user attribute that is equal to or greater than the maximum number of threads that are allocated at one time on the system. For example, you could increase the value of the max_threads_per_user attribute to 512.

On a very busy server with sufficient memory or an Internet server, increase the value of the max_threads_per_user attribute to 4096.

Setting the value of the max_threads_per_user attribute to 0 (zero) will remove the limit on threads.

If you specify a value of 0 (zero) for the max_threads_per_user attribute, there is no limit on threads.

See Section 3.6 for information about modifying kernel subsystem attributes.

5.2    Tuning Program Size Limits

If you are running a very large application, you may need to increase the values of the proc subsystem attributes that control program size limits. Some large programs and large-memory processes may not run unless you modify the default values of these attributes.

The following sections describe how to perform the following tasks:

• Increase the maximum size of a user process stack (Section 5.2.1)

• Increase the maximum size of a user process data segment (Section 5.2.2)

5.2.1    Increasing the Size of a User Process Stack

The proc subsystem attributes per_proc_stack_size and max_per_proc_stack_size specify the default and maximum sizes of a user process stack. Some large programs and large-memory processes may not run unless you increase the default value of these attributes.

Performance Benefit and Tradeoff

Increasing the default and maximum sizes of a user process stack enables very large applications to run.

You cannot modify the per_proc_stack_size max_per_proc_stack_size attributes without rebooting the system.

When to Tune

If you are running a large program or a large-memory process, or if you receive Cannot grow stack messages, increase the default and maximum sizes of a user process stack.

Recommended Values

The default value of the per_proc_stack_size attribute is 8388608 bytes. The default value of the max_per_proc_stack_size attribute is 33554432 bytes. Choose values that are significantly less than the address space limit. See Section 5.3 for information.

See Section 3.6 for information about modifying kernel subsystem attributes.

5.2.2    Increasing the Size of a User Process Data Segment

The proc subsystem attributes per_proc_data_size and max_per_proc_data_size specify the default and maximum sizes of a user process data segment. Some large programs and large-memory processes may not run unless you increase the default values of these attributes.

Performance Benefit and Tradeoff

Increasing the default and maximum sizes of a user process data segment enables very large applications to run.

You cannot modify the per_proc_data_size and max_per_proc_data_size attributes without rebooting the system.

When to Tune

You may need to increase the values of the per_proc_data_size and max_per_proc_data_size attributes if you are running a large program or a large-memory process, if you receive an Out of process memory message, or the system is an Internet server.

Recommended Values

The default value of the per_proc_data_size is 1342177281 bites. The default value of the max_per_proc_data_size is 1 GB (1073741824 bytes). Choose values that are significantly less than the address space limit. See Section 5.3 for information.

If you have an Internet server, increase the value of the max_per_proc_data_size attribute to 10 GB (10737418240 bytes).

See Section 3.6 for information about modifying kernel subsystem attributes.

5.3    Tuning Address Space Limits

The proc subsystem attributes per_proc_address_space and max_per_proc_address_space specify the default and maximum amount of user process address space (number of valid virtual regions).

Performance Benefit and Tradeoff

Increasing the address space limit enables large programs to run, improves the performance of memory-intensive applications. However, this causes a small increase in the demand for memory.

You cannot modify the per_proc_address_space and max_per_proc_address_space attributes without rebooting the system.

When to Tune

You may want to increase the address space limit if you are running a memory-intensive process, or if the system is an Internet server.

Recommended Values

The default value for the per_proc_address_space and max_per_proc_address_space attributes is 4 GB (4294967296 bytes).

If you have an Internet server, increase the value of the max_per_proc_address_space attribute to 10 GB (10737418240 bytes).

See Section 3.6 for information about modifying kernel attributes.

5.4    Tuning Interprocess Communication Limits

Interprocess communication (IPC) is the exchange of information between two or more processes. Some examples of IPC include messages, shared memory, semaphores, pipes, signals, process tracing, and processes communicating with other processes over a network.

The Tru64 UNIX operating system provides the following facilities for interprocess communication:

• Pipes -- See the Guide to Realtime Programming for information about pipes.

• Signals -- See the Guide to Realtime Programming for information.

• Sockets -- See the Network Programmer's Guide for information.

• Streams -- See the Programmer's Guide: STREAMS for information.

• X/Open Transport Interface (XTI) -- See the Network Programmer's Guide for information.

If you are running processes that are memory-intensive, you may want to increase the values of some ipc subsystem attributes.

Table 5-2 describes the guidelines for increasing IPC limits and lists the performance benefits as well as tradeoffs.

Table 5-2:  IPC Limits Tuning Guidelines

The following sections describe how to tune some System V attributes. See sys_attrs_ipc(5) for information about additional IPC subsystem attributes.

5.4.1    Increasing the Maximum Size of a System V Message

The ipc subsystem attribute msg_max specifies the maximum size of a System V message that an application can receive.

Performance Benefit and Tradeoff

Increasing the value of the msg_max attribute, may improve the performance of applications that can benefit from a System V message size that is larger than the default value. However, increasing this value will consume memory.

You cannot modify the msg_max attribute without rebooting the system.

When to Tune

If your applications can benefit from setting the default maximum size of a System V message to a value that is larger than 8192 bytes, you may want to increase the value of the msg_max attribute.

Recommended Values

The default value of the msg_max attribute is 8192 bytes (1 page).

See Section 3.6 for information about modifying kernel subsystem attributes.

5.4.2    Increasing the Maximum Size of a System V Message Queue

The ipc subsystem attribute msg_mnb, specifies the maximum number of bytes that can be in a System V message queue at one time.

A process cannot send a message to a queue if the number of bytes in the queue is greater than the limit specified by the msg_mnb attribute. When the limit is reached, the process sleeps and waits for this condition to be resolved.

Performance Benefit and Tradeoff

Increasing the value of the msg_mnb attribute may improve performance for applications that can benefit from a System V message queue that is larger than the default size. However, increasing this value will consume memory.

You cannot modify the msg_mnb attribute without rebooting the system.

When to Tune

You can track the use of IPC facilities with the ipcs -a command (see ipcs(1)). By looking at the current number of bytes and message headers in the queues, you can then determine whether you need to tune the System V message queue to diminish waiting.

Recommended Values

The default value of the msg_mnb attribute is 16384 bytes.

See Section 3.6 for information about modifying kernel subsystem attributes.

5.4.3    Increasing the Maximum Number of Messages on a System V Queue

The ipc subsystem attribute msg_tql specifies the maximum number of messages that can be on a System V message queue; that is, the total number of messages that can be outstanding in the system.

Performance Benefit and Tradeoff

Increasing the value of the msg_tql attribute may improve the performance of applications that benefit from increasing the number of outstanding messages to a value that is larger than the default value. However, increasing the value of this attribute will consume memory.

You cannot modify the msg_tql attribute without rebooting the system.

When to Tune

You may want to increase the value of the msg_tql attribute if your applications can benefit from increasing the maximum number of outstanding messages to a value than is larger than 40.

You can track the use of IPC facilities with the ipcs -a command (see ipcs(1)). By looking at the current number of bytes and message headers in the queues, you can then determine whether you need to tune the System V message queue to diminish waiting.

Recommended Values

The default value of the msg_tql is 40.

See Section 3.6 for information about modifying kernel subsystem attributes.

5.4.4    Increasing the Maximum Size of a System V Shared Memory Region

The ipc subsystem attribute shm_max specifies the maximum size of a single System V shared memory region.

Performance Benefit and Tradeoff

Increasing the value of the shm_max attribute may improve the performance of memory-intensive applications that can benefit from a large System V shared memory region. However, increasing the value of the shm_max attribute will increase the demand for memory.

You cannot modify the shm_max attribute without rebooting the system.

When to Tune

If your applications are memory-intensive and can benefit from a System V shared memory region that is larger than the default value of 512 pages, you may want to increase the value of the shm_max attribute.

Recommended Values

The default value of the shm_max attribute is 4194304 bytes (512 pages).

See Section 3.6 for information about modifying kernel subsystem attributes.

5.4.5    Increasing the Maximum Number of Shared Memory Regions Attached to a Process

The ipc subsystem attribute shm_seg specifies the maximum number of System V shared memory regions that can be attached to a single process at any point in time.

As a design consideration, consider whether you will get better performance by using threads instead of shared memory.

Performance Benefit and Tradeoff

Increasing the number of System V shared memory regions that can be attached to a single process may improve the performance of applications that attach many shared memory regions.

Increasing the value of the shm_seg attribute will consume memory if the process attaches many shared memory regions.

You cannot modify the shm_seg attribute without rebooting the system.

When to Tune

You may want to increase the value of the shm_seg attribute if a process' attempt to attach a shared memory region exceeds the limit (the shmat function returns an EMFILE error).

Recommended Values

The default value of the shm_seg is 32.

See Section 3.6 for information about modifying kernel subsystem attributes.

5.4.6    Modifying Shared Page Table Sharing

Third-level page table sharing occurs when the size of a System V shared memory segment, as created by the shmget function, is equal to or larger than the value of the ipc subsystem attribute ssm_threshold.

Performance Benefit and Tradeoff

Increasing the shared page table limit restricts shared page tables to applications that create shared memory segments larger than 8 MB. However, this will increase the demand for memory.

You can disable page table sharing, if your applications cannot use shared page tables.

You can modify the ssm_threshold attribute without rebooting the system.

When to Tune

If you want to restrict page table sharing to applications that create shared memory segments larger than 8 MB, increase the value of the ssm_threshold attribute.

If your applications cannot use shared pages tables because of alignment restrictions, you may want to disable the sharing of page tables.

Recommended Values

The default value of the ssm_threshold attribute is 8 MB (8388608 bytes).

Setting the ssm_threshold attribute to 0 (zero) will disable the use of segmented shared memory.

See Section 3.6 for information about modifying kernel subsystem attributes.

5.5    Tuning the Open File Limits

The following sections describe how to perform the following tasks:

• Increase the maximum number of open files (Section 5.5.1).

• Increase the maximum number of open file descriptors (Section 5.5.2).

5.5.1    Increasing the Maximum Number of Open Files

The kernel data structure for an open file is called a vnode. These are used by all file systems. The number of vnodes determines the number of open files. The allocation and deallocation of vnodes is handled dynamically by the operating system.

The vfs subsystem attribute max_vnodes specifies the size of the vnode cache, which is always equal to or more than the maximum number of open files in the system. You may need to increase the default value of this attribute to increase the maximum number of open files. Note that you can also accomplish this task by increasing the value of the proc subsystem attribute maxusers. See Section 5.1 for information.

Performance Benefit and Tradeoff

Increasing the size of the vnode cache can improve the performance of applications that require many open files, but it will also consume memory.

You can modify the max_vnodes attribute without rebooting the system.

When to Tune

If your applications require many open files or you receive a message indicating you are out of vnodes, increase the default value of the max_vnodes attribute.

Recommended Values

The default value of the max_vnodes attribute is 5 percent of memory.

See Section 3.6 for information about modifying kernel subsystem attributes.

5.5.2    Increasing the Maximum Number of Open File Descriptors

You may want to increase the maximum number of open file descriptors for all processes or for a specific application. The proc subsystem attributes open_max_soft and open_max_hard control the maximum system-wide number of open file descriptors for each process.

The open file descriptor limits prevent runaway allocations, such as allocations within a loop that cannot be exited because of an error condition, from consuming all of the available file descriptors. If a process reaches the open_max_soft limit, a warning message is issued. If a process reaches the open_max_hard limit, the process is stopped.

Performance Benefit and Tradeoff

Improves the performance of applications that open many files.

You cannot modify the open_max_soft and open_max_hard attributes without rebooting the system.

When to Tune

If you have an application that requires many open files, you can increase the open file descriptor limit by increasing the values of the open_max_soft and open_max_hard attributes. However, increasing the open file descriptor limit may cause runaway allocations.

Recommended Values

The default value of the open_max_soft and open_max_hard attributes is 4096, which is the maximum system-wide value that you can set in the /etc/sysconfigtab file.

If you have an application that requires many open files, you can increase the open file descriptor limit only for that application, instead of increasing the system-wide limit. To enable extended (64 KB) file descriptors for a specific application, follow these steps:

1. Set the setsysinfo system call's SSI_FD_NEWMAX operation parameter to 1, which sets the utask bit, enables up to 65,536 (64 KB) open file descriptors, and raises the process's hard file limit to 64 KB. This setting is inherited by any child process. See setsysinfo(2) for more information.

2. Set the process's file descriptor soft limit to a value that is more than 4096 (the default value) by using the setrlimit function as shown in the following code fragment:

   #include <sys/resource.h>
   struct rlimit *rlp;
    
   rlp->rlim_cur = 6000;
   rlp->rlim_max = 6000;
   setrlimit(RLIMIT_NOFILE, rlp);
   

   This setting is inherited by any child process. See setrlimit(2) for more information.

3. This step is required only for applications that use the select function's fd_set parameter, which points to an I/O descriptor set (and a FD_CLR, FD_ISSET, FD_SET, or FD_ZERO macro) and can modify an I/O descriptor set. If you meet these qualifications, you can use one of two procedures, one that enables a static definition of the maximum number of file descriptors or one that enables a dynamic definition:

   • Static definition:

     Override the default value of 4096 for FD_SETSIZE in the <sys/select.h> header file by specifying the maximum value of 65536. You must specify this value before you include the <sys/time.h> header file (which also includes the <sys/select.h> header file) in the code, as follows:

     #define FD_SETSIZE 65536
     #include <sys/time.h>
     

     This setting is not inherited by child processes; therefore, FD_SETSIZE must be set explicitly in the code for each child process that requires 64 KB file descriptors.

   • Dynamic definition:

     Instead of using statically defined fd_set structures, you can use fd_set pointers in conjunction with a malloc function, which provides forward compatibility with any future changes to the maximum file descriptor limit. For example:

     fd_set *fdp;
      
     fdp = (fd_set *) malloc(         
     (fds_howmany(max_fds,FD_NFDBITS))*sizeof(fd_mask));
     

     The value for max_fds is the number of file descriptors to be manipulated. It is recommended that you use the file descriptor soft limit for this value. All other keywords are defined in the <sys/select.h> header file. The following code segment shows this choice:

     #include <sys/time.h>         
     #include <sys/resource.h>
      
     my_program()         
     {
     fd_set *fdp;        
     struct rlimit rlim;
     int max_fds;
      
     getrlimit(RLIMIT_NOFILE, &rlim;);         
     max_fds = rlim.rlim_cur;
      
     fdp = (fd_set *) malloc(
     (fds_howmany(max_fds,FD_NFDBITS))*sizeof(fd_mask));
      
     FD_SET(2, fdp);
      
     for (;;) {         
     switch(select(max_fds, (fd_set *)0, fdp, (fd_set
     *)0,
     struct timeval *)0)) {     
     ...
     }
     

In addition, the vfs subsystem attribute max_vnodes must be set high enough for the needs of any application that requires a high number of descriptors. The max_vnodes attribute specifies the size of the vnode cache, and is set to 5 percent of system memory by default. See Section 5.5.1 for more information.

To disable support for up to 64 KB file descriptors for an application, set the setsysinfo system call's SSI_FD_NEWMAX operation parameter to 0, which disables the utask bit and returns the hard file limit to the default maximum of 4096 open file descriptors. However, if the process is using more than 4096 file descriptors, the setsysinfo system call will return an EINVAL error. In addition, if a calling process's hard or soft limit exceeds 4096, the limit is set to 4 KB after the call is successful. This setting is inherited by any child process.