Title and Copyright Information
 
About This Manual
Audience
New and Changed Features
Organization
Related Documents
Reader's Comments
Conventions
 
1    Introduction to Performance and Availability
1.1    Performance Terminology and Concepts
1.2    High-Performance Configurations
1.2.1    CPU Resources
1.2.2    Memory Resources
1.2.3    Disk Storage
1.2.3.1    RAID Technology
1.2.3.2    SCSI Concepts
1.2.3.2.1    Data Paths
1.2.3.2.2    Transmission Methods
1.2.3.2.3    SCSI Bus Speeds
1.2.3.2.4    SCSI Bus Length and Termination
1.2.4    Network Subsystem
1.3    High-Availability Configurations
 
2    Planning a High-Performance and High-Availability Configuration
2.1    Identifying a Resource Model for Your Workload
2.2    Identifying Performance and Availability Goals
2.3    Choosing System Hardware
2.3.1    CPU Configuration
2.3.2    Memory and Swap Space Configuration
2.3.2.1    Determining Your Physical Memory Requirements
2.3.2.2    Choosing a Swap Space Allocation Mode
2.3.2.3    Determining Swap Space Requirements
2.3.3    I/O Bus Slot Capacity
2.3.4    Support for High-Performance Disk Storage
2.3.5    Support for High-Performance Network
2.4    Choosing Disk Storage Hardware
2.4.1    Fast Disks
2.4.2    Solid-State Disks
2.4.3    Devices with Wide Data Paths
2.4.4    High-Performance Host Bus Adapters
2.4.5    DMA Host Bus Adapters
2.4.6    RAID Controllers
2.4.7    Fibre Channel
2.4.8    Prestoserve
2.5    Choosing How to Manage Disks
2.5.1    Using a Shared Pool of Storage for Flexible Management
2.5.2    Striping Data or Disks to Distribute I/O
2.5.3    Using Parity RAID to Improve Disk Performance
2.6    Choosing a High-Availability Configuration
2.6.1    Using a Cluster for System Availability
2.6.2    Using RAID for Disk Data Availability
2.6.3    Using Redundant Networks
2.6.4    Using Redundant Power Supplies and Systems
 
3    Monitoring Systems and Diagnosing Performance Problems
3.1    Obtaining Information About System Events
3.1.1    Using Event Manager
3.1.2    Using DECevent
3.2    Using System Accounting and Disk Quotas
3.3    Continuously Monitoring Performance
3.3.1    Using Performance Manager
3.3.2    Using Performance Visualizer
3.4    Gathering Performance Information
3.5    Profiling and Debugging Kernels
3.6    Accessing and Modifying Kernel Subsystems
3.6.1    Displaying the Subsystems Configured in the Kernel
3.6.2    Displaying Current Subsystem Attribute Values
3.6.3    Displaying Minimum and Maximum Attribute Values
3.6.4    Modifying Attribute Values at Run Time
3.6.5    Modifying Attribute Values at Boot Time
3.6.6    Permanently Modifying Attribute Values
3.6.7    Displaying and Modifying Kernel Variables by Using the dbx Debugger
 
4    Improving System Performance
4.1    Steps for Configuring and Tuning Systems
4.2    Tuning Special Configurations
4.2.1    Tuning Internet Servers
4.2.2    Tuning Large-Memory Systems
4.2.3    Tuning NFS Servers
4.3    Checking the Configuration by Using the sys_check Utility
4.4    Solving Common Performance Problems
4.4.1    Application Completes Slowly
4.4.2    Insufficient Memory or Excessive Paging
4.4.3    Insufficient Swap Space
4.4.4    Processes Swapped Out
4.4.5    Insufficient CPU Cycles
4.4.6    Disk Bottleneck
4.4.7    Poor Disk I/O Performance
4.4.8    Poor AdvFS Performance
4.4.9    Poor UFS Performance
4.4.10    Poor NFS Performance
4.4.11    Poor Network Performance
4.5    Using the Advanced Tuning Guidelines
 
5    Tuning System Resource Allocation
5.1    Tuning Process Limits
5.1.1    Increasing System Tables and Data Structures
5.1.2    Increasing the Maximum Number of Processes
5.1.3    Increasing the Maximum Number of Threads
5.2    Tuning Program Size Limits
5.2.1    Increasing the Size of a User Process Stack
5.2.2    Increasing the Size of a User Process Data Segment
5.3    Tuning Address Space Limits
5.4    Tuning Interprocess Communication Limits
5.4.1    Increasing the Maximum Size of a System V Message
5.4.2    Increasing the Maximum Size of a System V Message Queue
5.4.3    Increasing the Maximum Number of Messages on a System V Queue
5.4.4    Increasing the Maximum Size of a System V Shared Memory Region
5.4.5    Increasing the Maximum Number of Shared Memory Regions Attached to a Process
5.4.6    Modifying Shared Page Table Sharing
5.5    Tuning the Open File Limits
5.5.1    Increasing the Maximum Number of Open Files
5.5.2    Increasing the Maximum Number of Open File Descriptors
 
6    Managing Memory Performance
6.1    Virtual Memory Operation
6.1.1    Physical Page Tracking
6.1.2    File System Buffer Cache Memory Allocation
6.1.2.1    Metadata Buffer Cache Memory Allocation
6.1.2.2    Unified Buffer Cache Memory Allocation
6.1.2.3    AdvFS Buffer Cache Memory Allocation
6.1.3    Process Memory Allocation
6.1.3.1    Process Virtual Address Space Allocation
6.1.3.2    Virtual Address to Physical Address Translation
6.1.3.3    Page Faults
6.1.4    Page Reclamation
6.1.4.1    Modified Page Prewriting
6.1.4.2    Reclaiming Memory by Paging
6.1.4.3    Reclaiming Memory by Swapping
6.2    Configuring Swap Space for High Performance
6.3    Gathering Memory Information
6.3.1    Monitoring Memory by Using the vmstat Command
6.3.2    Monitoring Memory by Using the ps Command
6.3.3    Monitoring Swap Space Usage by Using the swapon Command
6.3.4    Monitoring the UBC by Using the dbx Debugger
6.4    Tuning to Provide More Memory to Processes
6.4.1    Reducing the Number of Processes Running Simultaneously
6.4.2    Reducing the Static Size of the Kernel
6.4.3    Decreasing the Borrowed Memory Threshold
6.4.4    Decreasing the Size of the AdvFS Buffer Cache
6.4.5    Decreasing the Memory for AdvFS Access Structures
6.4.6    Decreasing the Size of the Metadata Buffer Cache
6.4.7    Decreasing the Size of the namei Cache
6.4.8    Increasing the Memory Reserved for Kernel malloc Allocations
6.5    Tuning Paging and Swapping Operation
6.5.1    Increasing the Paging Threshold
6.5.2    Increasing the Rate of Swapping
6.5.3    Decreasing the Rate of Swapping
6.5.4    Enabling Aggressive Task Swapping
6.5.5    Limiting the Resident Set Size to Avoid Swapping
6.5.6    Increasing Modified Page Prewriting
6.5.7    Decreasing Modified Page Prewriting
6.5.8    Increasing the Size of the Page-In and Page-Out Clusters
6.5.9    Increasing the Swap I/O Queue Depth for Page Ins and Swap Outs
6.5.10    Decreasing the Swap I/O Queue Depth for Page Ins and Swap Outs
6.5.11    Increasing the Swap I/O Queue Depth for Page Outs
6.5.12    Decreasing the Swap I/O Queue Depth for Page Outs
6.6    Reserving Physical Memory for Shared Memory
6.6.1    Tuning the Kernel to Use Granularity Hints
6.6.2    Modifying Applications to Use Granularity Hints
 
7    Managing CPU Performance
7.1    Gathering CPU Performance Information
7.1.1    Monitoring CPU Usage by Using the ps Command
7.1.2    Monitoring CPU Statistics by Using the vmstat Command
7.1.3    Monitoring the Load Average by Using the uptime Command
7.1.4    Checking CPU Usage by Using the kdbx Debugger
7.1.5    Checking Lock Usage by Using the kdbx Debugger
7.2    Improving CPU Performance
7.2.1    Adding Processors
7.2.2    Using the Class Scheduler
7.2.3    Prioritizing Jobs
7.2.4    Scheduling Jobs at Offpeak Hours
7.2.5    Stopping the advfsd Daemon
7.2.6    Using Hardware RAID to Relieve the CPU of I/O Overhead
 
8    Managing Disk Storage Performance
8.1    Guidelines for Distributing the Disk I/O Load
8.2    Monitoring the Distribution of Disk I/O
8.3    Displaying Disk Usage by Using the iostat Command
8.4    Managing LSM Performance
8.4.1    LSM Features
8.4.2    Basic LSM Disk, Disk Group, and Volume Guidelines
8.4.2.1    Initializing LSM Disks as Sliced Disks
8.4.2.2    Sizing the rootdg Disk Group
8.4.2.3    Sizing Private Regions
8.4.2.4    Making Private Regions in a Disk Group the Same Size
8.4.2.5    Organizing Disk Groups
8.4.2.6    Mirroring the Root File System
8.4.2.7    Mirroring Swap Devices
8.4.2.8    Saving the LSM Configuration
8.4.3    LSM Mirrored Volume Configuration Guidelines
8.4.3.1    Placing Mirrored Plexes on Different Disks and Buses
8.4.3.2    Using Multiple Plexes in a Mirrored Volume
8.4.3.3    Choosing a Read Policy for a Mirrored Volume
8.4.3.4    Using a Symmetrical Plex Configuration
8.4.3.5    Using Hot Sparing for Mirrored Volumes
8.4.4    Dirty-Region Logging Configuration Guidelines
8.4.4.1    Configuring Log Plexes
8.4.4.2    Using the Correct Log Size
8.4.4.3    Placing Logging Subdisks on Infrequently Used Disks
8.4.4.4    Using Solid-State Disks for DRL Subdisks
8.4.4.5    Using a Nonvolatile Write-Back Cache for DRL
8.4.5    LSM Striped Volume Configuration Guidelines
8.4.5.1    Increasing the Number of Disks in a Striped Volume
8.4.5.2    Distributing Striped Volume Disks Across Different Buses
8.4.5.3    Choosing the Correct LSM Stripe Width
8.4.6    LSM RAID 5 Configuration Guidelines
8.4.6.1    Using RAID 5 Logging
8.4.6.2    Using the Appropriate Strip Width
8.4.6.3    Using Hot Sparing for RAID 5 Volumes
8.4.7    Gathering LSM Information
8.4.7.1    Displaying Configuration Information by Using the volprint Utility
8.4.7.2    Monitoring Performance Statistics by Using the volstat Utility
8.4.7.3    Tracking Operations by Using the voltrace Utility
8.4.7.4    Monitoring Events by Using the volwatch Script
8.4.7.5    Monitoring Events by Using the volnotify Utility
8.5    Managing Hardware RAID Subsystem Performance
8.5.1    Hardware RAID Features
8.5.2    Hardware RAID Products
8.5.3    Hardware RAID Configuration Guidelines
8.5.3.1    Distributing Storage Set Disks Across Buses
8.5.3.2    Using Disks with the Same Data Capacity
8.5.3.3    Choosing the Correct Hardware RAID Stripe Size
8.5.3.4    Mirroring Striped Sets
8.5.3.5    Using a Write-Back Cache
8.5.3.6    Using Dual-Redundant Controllers
8.5.3.7    Using Spare Disks to Replace Failed Disks
8.6    Managing CAM Performance
 
9    Managing File System Performance
9.1    Gathering File System Information
9.1.1    Displaying File System Disk Space
9.1.2    Checking the namei Cache with the dbx Debugger
9.2    Tuning File Systems
9.2.1    Increasing the Size of the namei Cache
9.2.2    Delaying vnode Deallocation
9.2.3    Delaying vnode Recycling
9.2.4    Increasing Memory for the UBC
9.2.5    Increasing the Borrowed Memory Threshold
9.2.6    Increasing the Minimum Size of the UBC
9.2.7    Improving Large File Caching Performance
9.2.8    Disabling File Read Access Time Flushing
9.2.9    Caching Only File System Metadata with Prestoserve
9.3    Managing Advanced File System Performance
9.3.1    AdvFS Features
9.3.2    AdvFS I/O Queues
9.3.3    AdvFS Access Structures
9.3.4    AdvFS Configuration Guidelines
9.3.4.1    Configuring File Domains
9.3.4.2    Configuring Filesets for High Performance
9.3.4.3    Distribute the AdvFS I/O Load
9.3.4.4    Improving the Transaction Log Performance
9.3.4.5    Forcing Synchronous Writes
9.3.4.6    Preventing Partial Data Writes
9.3.4.7    Enabling Direct I/O
9.3.4.8    Configuring an AdvFS root File system
9.3.4.9    Striping Files
9.3.4.10    Using AdvFS Quotas
9.3.4.11    Consolidating I/O Transfers
9.3.5    Gathering AdvFS Information
9.3.5.1    Monitoring AdvFS Performance Statistics by Using the advfsstat Command
9.3.5.2    Identifying Disks in an AdvFS File Domain by Using the advscan Command
9.3.5.3    Checking AdvFS File Domains by Using the showfdmn Command
9.3.5.4    Displaying AdvFS File Information by Using the showfile Command
9.3.5.5    Displaying the AdvFS Filesets in a File Domain by Using the showfsets Command
9.3.5.6    Monitoring the Bitmap Metadata Table
9.3.6    Tuning AdvFS
9.3.6.1    Increasing the Size of the AdvFS Buffer Cache
9.3.6.2    Increasing the Number of AdvFS Buffer Hash Chains
9.3.6.3    Increasing the Memory for Access Structures
9.3.6.4    Increasing Data Cached in the Ready Queue
9.3.6.5    Increasing the AdvFS Smooth Sync Cache Timeout Value
9.3.6.6    Specifying the Maximum Number of I/O Requests on the Device Queue
9.3.6.7    Disabling the Flushing of Modified mmapped Pages
9.3.7    Improving AdvFS Performance
9.3.7.1    Defragmenting a File Domain
9.3.7.2    Decreasing the I/O Transfer Size
9.3.7.3    Moving the Transaction Log
9.3.7.4    Balancing a Multivolume File Domain
9.3.7.5    Migrating Files Within a File Domain
9.4    Managing UFS Performance
9.4.1    UFS Configuration Guidelines
9.4.1.1    Modifying the File System Fragment and Block Sizes
9.4.1.2    Reducing the Density of inodes
9.4.1.3    Allocating Blocks Sequentially
9.4.1.4    Increasing the Number of Blocks Combined for a Cluster
9.4.1.5    Using MFS
9.4.1.6    Using UFS Disk Quotas
9.4.1.7    Increasing the Number of UFS and MFS Mounts
9.4.2    Gathering UFS Information
9.4.2.1    Displaying UFS Information by Using the dumpfs Command
9.4.2.2    Monitoring UFS Clustering by Using the dbx Debugger
9.4.2.3    Checking the Metadata Buffer Cache by Using the dbx Debugger
9.4.3    Tuning UFS
9.4.3.1    Increasing the Size of the Metadata Buffer Cache
9.4.3.2    Increasing the Size of the Metadata Hash Chain Table
9.4.3.3    Increasing the UFS Smooth Sync Cache Timeout Value
9.4.3.4    Delaying UFS Cluster Flushing
9.4.3.5    Increasing the Number of Blocks Combined for Read-Ahead
9.4.3.6    Increasing the Number of Blocks Combined for a Cluster
9.4.3.7    Defragmenting a File System
9.5    Managing NFS Performance
9.5.1    Gathering NFS Information
9.5.1.1    Displaying NFS Information by Using the nfsstat Command
9.5.1.2    Displaying Idle Thread Information by Using the ps Command
9.5.2    Improving NFS Performance
9.5.2.1    Using Prestoserve to Improve NFS Server Performance
9.5.2.2    Configuring Server Threads
9.5.2.3    Configuring Client Threads
9.5.2.4    Modifying Cache Timeout Limits
9.5.2.5    Decreasing Network Timeouts
9.5.2.6    Using NFS Protocol Version 3
 
10    Managing Network Performance
10.1    Gathering Network Information
10.1.1    Monitoring Network Statistics by Using the netstat Command
10.1.2    Checking Socket Listen Queue Statistics by Using the sysconfig Command
10.2    Tuning the Network Subsystem
10.2.1    Improving the Lookup Rate for TCP Control Blocks
10.2.2    Increasing the Number of TCP Hash Tables
10.2.3    Tuning the TCP Socket Listen Queue Limits
10.2.4    Increasing the Number of Outgoing Connection Ports
10.2.5    Modifying the Range of Outgoing Connection Ports
10.2.6    Disabling Use of a PMTU
10.2.7    Increasing the Number of IP Input Queues
10.2.8    Enabling mbuf Cluster Compression
10.2.9    Enabling TCP Keepalive Functionality
10.2.10    Improving the Lookup Rate for IP Addresses
10.2.11    Decreasing the TCP Partial-Connection Timeout Limit
10.2.12    Decreasing the TCP Connection Context Timeout Limit
10.2.13    Decreasing the TCP Retransmission Rate
10.2.14    Disabling Delaying the Acknowledgment of TCP Data
10.2.15    Increasing the Maximum TCP Segment Size
10.2.16    Increasing the Transmit and Receive Buffers for a TCP Socket
10.2.17    Increasing the Transmit and Receive Buffers for a UDP Socket
10.2.18    Increasing the Size of the ARP Table
10.2.19    Increasing the Maximum Size of a Socket Buffer
10.2.20    Preventing Dropped Input Packets
 
11    Managing Application Performance
11.1    Gathering Profiling and Debugging Information
11.2    Improving Application Performance
11.2.1    Using the Latest Operating System Patches
11.2.2    Using the Latest Version of the Compiler
11.2.3    Using Parallelism
11.2.4    Optimizing Applications
11.2.5    Using Shared Libraries
11.2.6    Reducing Application Memory Requirements
11.2.7    Controlling Memory Locking
 
Glossary
 
Figures
1-1    Moving Instructions and Data Through the Memory Hardware
1-2    Physical Memory Usage
1-3    Configuration with Potential Points of Failure
1-4    Fully Redundant Cluster Configuration
6-1    UBC Memory Allocation
6-2    Memory Allocation During High File System Activity and No Paging Activity
6-3    Memory Allocation During Low File System Activity and High Paging Activity
6-4    Virtual Address Space Usage
6-5    Virtual-to-Physical Address Translation
6-6    Paging and Swapping Attributes
6-7    Paging Operation
9-1    AdvFS I/O Queues
 
Tables
1-1    Memory Management Hardware Resources
1-2    RAID Level Performance and Availability Comparison
1-3    SCSI Bus Speeds
1-4    SCSI Bus and Segment Lengths
2-1    Resource Models and Possible Configuration Solutions
2-2    High-Performance System Hardware Options
2-3    High-Performance Disk Storage Hardware Options
2-4    High-Performance Disk Storage Configuration Solutions
2-5    High-Availability Configurations
3-1    Tools for Continuous Performance Monitoring
3-2    Kernel Profiling and Debugging Tools
4-1    Internet Server Tuning Guidelines
4-2    Large-Memory System Tuning Guidelines
4-3    NFS Server Tuning Guidelines
4-4    Advanced Tuning Guidelines
5-1    Default Values for the maxusers Attribute
5-2    IPC Limits Tuning Guidelines
6-1    Default Values for vm_page_free_target Attribute
6-2    Virtual Memory and UBC Monitoring Tools
6-3    Memory Resource Tuning Guidelines
6-4    Paging and Swapping Tuning Guidelines
7-1    CPU Monitoring Tools
7-2    Primary CPU Performance Improvement Guidelines
8-1    Disk I/O Distribution Monitoring Tools
8-2    LSM Disk, Disk Group, and Volume Configuration Guidelines
8-3    LSM Mirrored Volume Guidelines
8-4    Dirty-Region Logging Guidelines
8-5    LSM Striped Volume Guidelines
8-6    LSM RAID 5 Volume Guidelines
8-7    LSM Monitoring Tools
8-8    Hardware RAID Subsystem Configuration Guidelines
9-1    General File System Tuning Guidelines
9-2    AdvFS Configuration Guidelines
9-3    AdvFS Monitoring Tools
9-4    AdvFS Tuning Guidelines
9-5    AdvFS Performance Improvement Guidelines
9-6    UFS Configuration Guidelines
9-7    UFS Monitoring Tools
9-8    UFS Tuning Guidelines
9-9    NFS Monitoring Tools
9-10    NFS Performance Guidelines
10-1    Network Monitoring Tools
10-2    Network Tuning Guidelines
11-1    Application Profiling and Debugging Tools
11-2    Application Performance Improvement Guidelines
 
Index