Title and Copyright Information
 
About This Manual
Audience
New and Changed Features
Organization
Related Documents
Reader's Comments
Conventions
 
1    Introduction to Performance and Availability
1.1    Using the Configuration and Tuning Methodology
1.2    Defining System Performance
1.3    Understanding High-Performance Configurations
1.3.1    CPU Resources
1.3.2    Memory Resources
1.3.3    Disk Storage Configuration
1.3.3.1    RAID Technology
1.3.3.2    SCSI Concepts
1.3.3.2.1    Data Paths
1.3.3.2.2    Transmission Methods
1.3.3.2.3    SCSI Bus Speeds
1.3.3.2.4    SCSI Bus Length and Termination
1.3.4    Network Subsystem
1.4    Understanding High-Availability Configurations
 
2    Planning a High-Performance and High-Availability Configuration
2.1    Identifying a Resource Model for Your Workload
2.2    Identifying Performance and Availability Goals
2.3    Choosing High-Performance System Hardware
2.3.1    CPU Configuration
2.3.2    Memory and Swap Space Configuration
2.3.2.1    Determining Your Physical Memory Requirements
2.3.2.2    Choosing a Swap Space Allocation Mode
2.3.2.3    Determining Swap Space Requirements
2.3.3    I/O Capacity
2.3.4    High-Performance Disk Storage Support
2.3.5    High-Performance Network Support
2.4    Choosing High-Performance Disk Storage Hardware
2.4.1    Fast Disks
2.4.2    Solid-State Disks
2.4.3    Devices with Wide Data Paths
2.4.4    High-Performance Host Bus Adapters
2.4.5    DMA Host Bus Adapters
2.4.6    RAID Controllers
2.4.7    Fibre Channel
2.4.8    Prestoserve
2.5    Choosing How to Manage Disks
2.5.1    Using a Shared Pool of Storage for Flexible Management
2.5.2    Striping Disks to Distribute I/O
2.5.3    Using Parity RAID to Improve Disk Performance
2.6    Choosing a High-Availability Configuration
2.6.1    Using a Cluster for System Availability
2.6.2    Using RAID for Disk Data Availability
2.6.3    Using Redundant Networks
2.6.4    Using Redundant Power Sources
 
3    Monitoring Systems and Diagnosing Performance Problems
3.1    Obtaining Information About System Events
3.2    Using System Accounting and Disk Quotas
3.3    Gathering Performance Information
3.4    Continuously Monitoring Performance
3.4.1    Using Performance Manager
3.4.2    Using Performance Visualizer
3.4.3    Using Digital Continuous Profiling Infrastructure
3.5    Profiling and Debugging Kernels
 
4    Improving System Performance
4.1    Tuning Special Configurations
4.1.1    Tuning Internet Servers
4.1.2    Tuning Large-Memory Systems
4.1.3    Tuning NFS Servers
4.2    Checking the Configuration by Using the sys_check Utility
4.3    Using the Advanced Tuning Recommendations
4.4    Modifying the Kernel
4.4.1    Displaying Kernel Subsystems
4.4.2    Displaying Current Attribute Values
4.4.3    Displaying Minimum and Maximum Attribute Values
4.4.4    Modifying Run-Time Attribute Values
4.4.5    Permanently Modifying Attribute Values
4.4.6    Displaying and Modifying Kernel Variables by Using the dbx Debugger
4.5    Solving Common Performance Problems
4.5.1    Application Completes Slowly
4.5.2    Excessive Memory Paging
4.5.3    Insufficient Swap Space
4.5.4    Insufficient CPU Cycles
4.5.5    Processes Swapped Out
4.5.6    Disk Bottleneck
4.5.7    Poor Disk I/O Performance
4.5.8    Poor AdvFS Performance
4.5.9    Poor UFS Performance
4.5.10    Poor NFS Performance
4.5.11    Poor Network Performance
 
5    Tuning System Resource Limits
5.1    Tuning Process Limits
5.2    Tuning Program Size Limits
5.3    Tuning Virtual Memory Limits
5.3.1    Increasing Address Space
5.3.2    Increasing the Number of Memory-Mapped Files
5.3.3    Increasing the Number of Pages with Individual Protections
5.4    Tuning Interprocess Communication Limits
5.4.1    Increasing the Maximum Size of a System V Message
5.4.2    Increasing the Maximum Size of a System V Message Queue
5.4.3    Increasing the Maximum Number of Messages on a System V Queue
5.4.4    Increasing the Maximum Size of a System V Shared Memory Region
5.4.5    Increasing the Minimum Size of a System V Shared Memory Segment
5.5    Tuning the Open File Limits
5.5.1    Increasing the Maximum Number of Open Files
5.5.2    Increasing the Maximum Number of Open File Descriptors
 
6    Managing Memory Performance
6.1    Understanding Memory Operation
6.1.1    Tracking Physical Pages
6.1.2    Allocating Memory to the File System Buffer Caches
6.1.2.1    Allocating Wired Memory to the Metadata Buffer Cache
6.1.2.2    Allocating Virtual Memory to the Unified Buffer Cache
6.1.2.3    Allocating Memory to the AdvFS Buffer Cache
6.1.3    Allocating Virtual Memory to Processes
6.1.3.1    Allocating Virtual Address Space to Processes
6.1.3.2    Translating Virtual Addresses to Physical Addresses
6.1.3.3    Page Faulting
6.1.4    Reclaiming Pages
6.1.4.1    Prewriting Modified Pages
6.1.4.2    Reclaiming Memory by Paging
6.1.4.3    Reclaiming Memory by Swapping
6.2    Configuring Swap Space for High Performance
6.3    Gathering Memory Information
6.3.1    Monitoring Memory by Using the ps Command
6.3.2    Monitoring Memory by Using the vmstat Command
6.3.3    Monitoring Swap Space Usage by Using the swapon Command
6.3.4    Monitoring Memory by Using the dbx Debugger
6.3.5    Monitoring the UBC by Using the dbx Debugger
6.4    Tuning to Provide More Memory to Processes
6.4.1    Reducing the Number of Processes Running Simultaneously
6.4.2    Reducing the Static Size of the Kernel
6.4.3    Reducing Dynamically Wired Memory
6.4.4    Decreasing the Memory Available to the UBC
6.4.5    Decreasing the Size of the AdvFS Buffer Cache
6.4.6    Decreasing the Size of the Metadata Buffer Cache
6.4.7    Decreasing the Size of the namei Cache
6.4.8    Increasing the Memory Reserved for Kernel Allocations
6.5    Tuning Paging and Swapping Operation
6.5.1    Changing the Rate of Swapping
6.5.2    Controlling Dirty Page Prewriting
6.5.3    Modifying the Size of the Page-In and Page-Out Clusters
6.5.4    Modifying the Swap I/O Queue Depth for Pageins and Swapouts
6.5.5    Modifying the Swap I/O Queue Depth for Pageouts
6.5.6    Increasing the Paging Threshold
6.5.7    Enabling Aggressive Task Swapping
6.6    Tuning to Reserve Physical Memory for Shared Memory
6.6.1    Tuning the Kernel to Use Granularity Hints
6.6.2    Modifying Applications to Use Granularity Hints
 
7    Managing CPU Performance
7.1    Gathering CPU Performance Information
7.1.1    Monitoring CPU Usage by Using the ps Command
7.1.2    Monitoring CPU Statistics by Using the vmstat Command
7.1.3    Monitoring the Load Average by Using the uptime Command
7.1.4    Checking CPU Usage by Using the kdbx Debugger
7.1.5    Checking Lock Usage by Using the kdbx Debugger
7.2    Improving CPU Performance
7.2.1    Adding Processors
7.2.2    Using the Class Scheduler
7.2.3    Prioritizing Jobs
7.2.4    Scheduling Jobs at Offpeak Hours
7.2.5    Stopping the advfsd Daemon
7.2.6    Using Hardware RAID to Relieve the CPU of I/O Overhead
 
8    Managing Disk Storage Performance
8.1    Distributing the Disk I/O Load
8.2    Gathering Basic Disk Information
8.2.1    Displaying Disk Usage with the iostat Command
8.2.2    Monitoring CAM by Using the dbx Debugger
8.3    Managing Logical Storage Manager Performance
8.3.1    Basic LSM Configuration Recommendations
8.3.1.1    Initializing LSM Disks as Sliced Disks
8.3.1.2    Sizing the rootdg Disk Group
8.3.1.3    Sizing Private Regions
8.3.1.4    Making Private Regions in a Disk Group the Same Size
8.3.1.5    Organizing Disks in Disk Groups
8.3.1.6    Choosing the Correct Number and Size of the Database and Log Copies
8.3.1.7    Distributing the Database and Logs Across Different Buses
8.3.2    LSM Mirrored Volume Configuration Recommendations
8.3.2.1    Mirroring Volumes Across Different Buses
8.3.2.2    Choosing a Read Policy for a Mirrored Volume
8.3.2.3    Using Multiple Plexes in a Mirrored Volume
8.3.2.4    Using a Symmetrical Configuration
8.3.2.5    Using Multiple BCL Subdisks
8.3.2.6    Using a Write-Back Cache with LSM
8.3.2.7    Sizing BCL Subdisks
8.3.2.8    Placing BCL Logging Subdisks on Infrequently Used Disks
8.3.2.9    Using Solid-State Disks for BCL Subdisks
8.3.3    LSM Striped Volume Configuration Recommendations
8.3.3.1    Increasing the Number of Disks in a Striped Volume
8.3.3.2    Distributing Striped Volume Subdisks Across Different Buses
8.3.3.3    Choosing the Correct Stripe Width
8.3.4    Gathering LSM Information
8.3.4.1    Displaying LSM Configuration Information by Using the volprint Utility
8.3.4.2    Monitoring LSM Performance Statistics by Using the volstat Utility
8.3.4.3    Tracking LSM Operations by Using the voltrace Utility
8.3.4.4    Monitoring LSM Events by Using the volwatch Script
8.3.4.5    Monitoring LSM by Using the dxlsm Graphical User Interface
8.3.5    Improving LSM Performance
8.4    Managing Hardware RAID Subsystem Performance
8.4.1    Distributing Storage Set Disks Across Buses
8.4.2    Using Disks with the Same Data Capacity
8.4.3    Choosing the Correct Stripe Size
8.4.4    Mirroring Striped Sets
8.4.5    Using a Write-Back Cache
8.4.6    Using Dual-Redundant Controllers
8.4.7    Using Spare Disks to Replace Failed Disks
8.5    Tuning CAM
 
9    Managing File System Performance
9.1    Gathering File System Information
9.1.1    Monitoring the Unified Buffer Cache
9.1.2    Checking the namei Cache with the dbx Debugger
9.2    Tuning File Systems
9.2.1    Increasing the Size of the namei Cache
9.2.2    Increasing the Size of the Hash Chain Table for the namei Cache
9.2.3    Increasing Memory for the UBC
9.2.4    Decreasing the Amount of Borrowed Memory
9.2.5    Increasing the Minimum Size of the UBC
9.2.6    Modifying the UBC Write Device Queue Depth
9.2.7    Controlling Large File Caching
9.2.8    Disabling File Read Access Time Flushing
9.2.9    Using Prestoserve to Cache Only File System Metadata
9.2.10    Increasing the Size of the Prestoserve Buffer Hash Table
9.2.11    Caching More Free vnodes
9.2.12    Increasing the Time vnodes Remain on the Free List
9.2.13    Delaying the Deallocation of vnodes
9.2.14    Accelerating the Deallocation of vnodes
9.2.15    Disabling vnode Deallocation
9.3    Managing Advanced File System Performance
9.3.1    Understanding AdvFS Operation
9.3.1.1    AdvFS I/O Queues
9.3.1.2    AdvFS Access Structures
9.3.2    AdvFS Configuration Guidelines
9.3.2.1    Using Multiple-Volume File Domains
9.3.2.2    Improving the Transaction Log Performance
9.3.2.3    Improving Bitmap Metadata Table Performance
9.3.2.4    Striping Files
9.3.2.5    Using AdvFS Quotas
9.3.3    Gathering AdvFS Information
9.3.3.1    Monitoring AdvFS Performance Statistics by Using the advfsstat Command
9.3.3.2    Identifying Disks in an AdvFS File Domain by Using the advscan Command
9.3.3.3    Checking AdvFS File Domains by Using the showfdmn Command
9.3.3.4    Displaying AdvFS File Information by Using the showfile Command
9.3.3.5    Displaying the AdvFS Filesets in a File Domain by Using the showfsets Command
9.3.4    Tuning AdvFS
9.3.4.1    Modifying the Size of the AdvFS Buffer Cache
9.3.4.2    Increasing the Number of AdvFS Buffer Hash Chains
9.3.4.3    Increasing the Memory Reserved for Access Structures
9.3.4.4    Defragmenting a File Domain
9.3.4.5    Increasing the Amount of Data Cached in the Ready Queue
9.3.4.6    Decreasing the Maximum Number of I/O Requests on the Device Queue
9.3.4.7    Decreasing the I/O Transfer Size
9.3.4.8    Disabling the Flushing of Modified mmapped Pages
9.3.4.9    Consolidating I/O Transfers
9.3.4.10    Forcing Synchronous Writes
9.3.4.11    Preventing Partial Data Writes
9.3.4.12    Moving the Transaction Log
9.3.4.13    Balancing a Multivolume File Domain
9.3.4.14    Migrating Files Within a File Domain
9.4    Managing UNIX File System Performance
9.4.1    UFS Configuration Guidelines
9.4.1.1    Modifying the UFS Fragment Size
9.4.1.2    Reducing the Density of inodes
9.4.1.3    Allocating Blocks Sequentially
9.4.1.4    Increasing the Number of Blocks Combined for a Cluster
9.4.1.5    Using a Memory File System
9.4.1.6    Using UFS Disk Quotas
9.4.2    Gathering UFS Information
9.4.2.1    Displaying UFS Information by Using the dumpfs Command
9.4.2.2    Monitoring UFS Clustering by Using the dbx Debugger
9.4.2.3    Checking the Metadata Buffer Cache by Using the dbx Debugger
9.4.3    Tuning UFS
9.4.3.1    Modifying the Size of the Metadata Buffer Cache
9.4.3.2    Increasing the Size of the Metadata Hash Chain Table
9.4.3.3    Defragmenting a File System
9.4.3.4    Delaying Full Write Buffer Flushing
9.4.3.5    Increasing the Number of Blocks Combined for Read Ahead
9.4.3.6    Increasing the Number of Blocks Combined for a Cluster
9.4.3.7    Modifying UFS Smooth Sync Caching
9.4.3.8    Increasing the Number of UFS and MFS Mounts
9.5    Managing NFS Performance
9.5.1    Gathering NFS Information
9.5.1.1    Displaying NFS Information by Using the nfsstat Command
9.5.1.2    Displaying Idle Thread Information by Using the ps Command
9.5.2    Improving NFS Performance
9.5.2.1    Using Prestoserve to Improve NFS Server Performance
9.5.2.2    Configuring Server Threads
9.5.2.3    Configuring Client Threads
9.5.2.4    Modifying Cache Timeout Limits
9.5.2.5    Decreasing Network Timeouts
9.5.2.6    Using NFS Protocol Version 3
 
10    Managing Network Performance
10.1    Gathering Network Information
10.1.1    Monitoring Network Statistics by Using the netstat Command
10.1.2    Checking Socket Listen Queue Statistics by Using the sysconfig Command
10.2    Tuning the Network Subsystem
10.2.1    Improving the Lookup Rate for TCP Control Blocks
10.2.2    Increasing the Number of TCP Hash Tables
10.2.3    Tuning the Socket Listen Queue Limits
10.2.4    Increasing the Number of Outgoing Connection Ports
10.2.5    Modifying the Range of Outgoing Connection Ports
10.2.6    Enabling TCP Keepalive Functionality
10.2.7    Improving the Lookup Rate for IP Addresses
10.2.8    Decreasing the Partial TCP Connection Timeout Limit
10.2.9    Decreasing the TCP Connection Context Timeout Limit
10.2.10    Decreasing the TCP Retransmission Rate
10.2.11    Disabling Delaying the Acknowledgment of TCP Data
10.2.12    Increasing the Maximum TCP Segment Size
10.2.13    Increasing the Transmit and Receive Socket Buffers
10.2.14    Increasing the Transmit and Receive Buffers for a UDP Socket
10.2.15    Allocating Sufficient Memory to the UBC
10.2.16    Disabling Use of a PMTU
10.2.17    Increasing the Size of the ARP Table
10.2.18    Increasing the Maximum Size of a Socket Buffer
10.2.19    Increasing the Number of IP Input Queues
10.2.20    Preventing Dropped Input Packets
10.2.21    Enabling mbuf Cluster Compression
10.2.22    Modifying the NetRAIN Retry Limit
10.2.23    Modifying the NetRAIN Monitoring Timer
 
11    Managing Application Performance
11.1    Gathering Profiling and Debugging Information
11.2    Improving Application Performance
11.2.1    Using the Latest Operating System Patches
11.2.2    Using the Latest Version of the Compiler
11.2.3    Using Parallelism
11.2.4    Optimizing Applications
11.2.5    Using Shared Libraries
11.2.6    Reducing Application Memory Requirements
11.2.7    Controlling Memory Locking
 
Glossary
 
Figures
1-1    Moving Instructions and Data Through the Memory Hardware
1-2    Physical Memory Usage
1-3    Configuration with Potential Points of Failure
1-4    Fully Redundant Cluster Configuration
6-1    UBC Memory Allocation
6-2    Memory Allocation During High File System Activity and No Paging Activity
6-3    Memory Allocation During Low File System Activity and High Paging Activity
6-4    Virtual Address Space Usage
6-5    Virtual-to-Physical Address Translation
6-6    Paging and Swapping Attributes - Default Values
6-7    Paging Operation
9-1    AdvFS I/O Queues
 
Tables
1-1    Memory Management Hardware Resources
1-2    RAID Level Performance and Availability Comparison
1-3    SCSI Bus Speeds
1-4    SCSI Bus and Segment Lengths
2-1    Resource Models and Possible Configuration Solutions
2-2    High-Performance System Hardware Options
2-3    High-Performance Disk Storage Hardware Options
2-4    High-Performance Disk Storage Configuration Solutions
2-5    High-Availability Configurations
3-1    Kernel Profiling and Debugging Tools
4-1    Internet Server Tuning Recommendations
4-2    Large-Memory System Tuning Recommendations
4-3    NFS Server Tuning Recommendations
4-4    Advanced Tuning Guidelines
5-1    Virtual Memory Limits Tuning Guidelines
5-2    IPC Limits Tuning Guidelines
6-1    Virtual Memory and UBC Monitoring Tools
6-2    Memory Resource Tuning Guidelines
6-3    Paging and Swapping Tuning Guidelines
7-1    CPU Monitoring Tools
7-2    Primary CPU Performance Improvement Guidelines
8-1    Basic Disk Monitoring Tools
8-2    LSM Disk, Disk Group, and Database Configuration Guidelines
8-3    LSM Database and Kernel Change Log Guidelines
8-4    LSM Mirrored Volume Guidelines
8-5    LSM Block-Change Logging Guidelines
8-6    LSM Striped Volume Guidelines
8-7    LSM Monitoring Tools
8-8    Hardware RAID Subsystem Configuration Guidelines
9-1    General File System Tuning Guidelines
9-2    AdvFS Configuration Guidelines
9-3    BMT Sizing Guidelines
9-4    AdvFS Monitoring Tools
9-5    AdvFS Tuning Guidelines
9-6    UFS Configuration Guidelines
9-7    UFS Monitoring Tools
9-8    UFS Tuning Guidelines
9-9    NFS Monitoring Tools
9-10    NFS Performance Guidelines
10-1    Network Monitoring Tools
10-2    Network Tuning Guidelines
11-1    Application Profiling and Debugging Tools
11-2    Application Performance Improvement Guidelines
 
Index