5    Optimizing Performance

AdvFS provides a number of ways to configure and tune your file system. Some of the tuning functions are available through the AdvFS GUI (see Chapter 6). The System Configuration and Tuning manual provides detailed information.

It is helpful to understand how your system will be used before you configure and tune your system. Be careful not to optimize toward a workload that is not easily understood or that may change.

5.1    Configuring AdvFS

You will obtain the best performance if you carefully plan your AdvFS configuration. You can control how you configure your file domains and how you allocate disks. You can turn on direct I/O to speed data transfer. You can make choices about transaction logging and file structure.

5.1.1    Configuring File Domains

There is a trade-off in using one large file domain instead of several smaller ones. Because each file domain has one transaction log, creating a single large file domain decreases maintenance complexity at the cost of putting a greater load on the log, which may become a bottleneck. See System Configuration and Tuning for more detailed information about allocating file domains effectively.

File domains that were created on operating systems prior to Version 5.0 do not have the structure necessary to provide large quota values and better performance for directories containing thousands of files. If either of these new features is important to you, update your file domains (see Section 2.3.3).

5.1.2    Configuring Volumes

You can add multiple volumes to an AdvFS file domain. This may improve performance because I/O processes can run in parallel. However, without LSM disk mirroring, it is inadvisable to add more than eight volumes. If you lose a volume, the entire domain becomes inaccessible. The risk of losing a volume, and thus losing access to your file domain, increases as the number of volumes increases.

In many cases, there is a significant performance advantage to dividing disks on different SCSI busses. See System Configuration and Tuning for more detailed information.

5.1.3    Improving Transaction Log Performance

Each file domain has a transaction log that keeps track of fileset activity for all filesets in the file domain. If the log resides on a congested disk or bus, or if the file domain contains many filesets, system performance can degrade.

You can monitor performance of the volume with Performance Manager (see Section 7.4) or the iostat utility. Do one of the following if the volume containing the log appears overloaded:

To move the transaction log to another volume:

  1. Use the showfdmn command to determine the location of the log. The letter L after the volume number indicates the volume on which the log resides.

  2. Use the switchlog command to move the log to another volume:

    switchlog domain_name new_volume_number

For example, to move the transaction log for the file domain january: 

# showfdmn january 
     Id              Date Created     LogPgs Version Domain Name
31bf51ba.0001be10 Wed Jun  9 16:24 1999  512       4 january
 
Vol  512-Blks    Free % Used Cmode Rblks Wblks Vol Name
  1L  1787904  885168    52%    on   128   128 /dev/disk/dsk0g
  2   1790096 1403872    22%    on   128   128 /dev/disk/dsk0h
     -------------------------
      3578000 2259040    37% 
# switchlog january 2
# showfdmn january 
     Id              Date Created     LogPgs Version Domain Name
31bf51ba.0001be10 Wed Jun  9 16:24 1999  512       4 january
 
Vol  512-Blks    Free % Used Cmode Rblks Wblks Vol Name
  1   1787904  885168    52%    on   128   128 /dev/disk/dsk0g
  2L  1790096 1395680    22%    on   128   128 /dev/disk/dsk0h
     -------------------------
      3578000 2250848    37%

Moving the transaction log may also be useful when you are using LSM and wish to increase reliability by placing your transaction log on a volume that is mirrored.

5.1.4    Improving Data Transfer Rate with Direct I/O

You can use direct I/O mode to synchronously read and write data from a file without copying the data into a cache (the normal AdvFS process). That is, when direct I/O is enabled for a file, read and write requests on it are executed to and from disk storage through direct memory access(similar to raw I/O), bypassing AdvFS caching. This may improve the speed of the I/O process for applications that access data only once.

Although direct I/O will handle I/O requests of any byte size, the best performance will occur when the requested byte size is aligned on file page boundaries and is evenly divisible into 8-kilobyte pages. Direct transfer from the user buffer to the disk is optimized in this configuration.

Direct I/O is particularly suited for files that are used exclusively by a database. However, if an application tends to access data multiple times, direct I/O can adversely impact performance because caching will not occur. As soon as you specify direct I/O, it takes precedence and any data already in the buffer cache for that file will automatically be flushed to disk.

To open a file for direct I/O, use the open() function and specify the O_DIRECTIO flag. If the file is already open for direct I/O or is in cached mode, the new mode will be direct I/O and will remain so until the last close of the file. Note that direct I/O, atomic write data logging, and mmapping are mutually exclusive modes. Therefore, if the file is already open for atomic write data logging or is mmapped, then calling the open function to initiate direct I/O will fail.

The fcntl() function can be used to determine whether the file is open in cached or in direct I/O mode. See fnctl(2) and open(2) or the Programmer's Guide for more information.

5.2    Tuning AdvFS

There are a number of things you can do to operate AdvFS more efficiently. You can defragment a file domain, balance a multivolume file domain to even the storage distribution, stripe files across disks to improve read/write performance, and migrate files to faster volumes. You can change caching attributes, I/O transfer parameters, and other AdvFS attributes. Detailed information about tuning is available in System Configuration and Tuning.

5.2.1    Increasing the AdvFS Buffer Cache

Caching data in the AdvFS buffer cache increases the throughput if the cached data is reused. Increasing the cache decreases the amount of memory available for the virtual memory system. You can specify the amount of physical memory that AdvFS uses for the buffer cache with the AdvfsCacheMaxPercent attribute (seeSystem Configuration and Tuning).

5.2.2    Defragmenting a File Domain

AdvFS attempts to store file data in contiguous blocks on a disk. This collection of contiguous blocks is called a file extent. If all data in a file is stored in contiguous blocks, that file has one file extent. However, as files grow, contiguous blocks on the disk may not be available to accommodate the new data, and the system will spread the file over discontiguous blocks. As a result, the file is fragmented on the disk and consists of multiple file extents. File fragmentation degrades the read/write performance because many disk addresses must be examined to access a file.

Figure 5-1:  Defragmenting a File Domain

The defragment utility reduces the amount of file fragmentation in a file domain by attempting to make the files more contiguous. Defragmentation, as illustrated in Figure 5-1, is an iterative, two-step process that operates on the file domain:

  1. Files are moved out of a region to create an area with contiguous, unallocated space.

  2. Fragmented files are written into a region that has more contiguous space so they are less fragmented.

In addition to making files contiguous so that the number of file extents is reduced, defragmenting a file domain often makes the free space on a disk more contiguous so files that are created later will also be less fragmented.

You can improve the efficiency of the defragment process by deleting any unneeded files in the file domain before running the defragment utility. Aborting the defragment process does not damage the file system. Files that have been defragmented remain in their new locations.

It is difficult to specify the load that defragmenting will place on a system. The time it takes to defragment a file domain depends on:

To defragment a file domain, use the AdvFS GUI (see Chapter 6) or, from the command line, enter the defragment command:

defragment domain_name

This functionality is also available from the SysMan menu.

The following restrictions apply to running the defragment command:

See defragment(8) for more information.

5.2.2.1    Choosing to Defragment

Running the defragment command may have a performance penalty, so it is a good idea to run the utility when file system activity is low. How fragmented you should let your file system become before running the utility depends on the size of the files and the number of extents. These are generally application dependent, so it is a good idea to monitor the number of extents (see defragment(8)) to see if elevated extent counts correlate with decreased application performance. In many cases, even a large, fairly fragmented file will show no noticeable decrease in performance because of fragmentation.

It is not necessary to run defragment:

If you want to determine the amount of file fragmentation that exists in a domain without altering the domain, run the defragment command with the -v and -n options. From the output you can determine how fragmented your file domain is and if it is a possible cause of poor system performance.

For example, to determine the fragmentation of the file domain staff_dmn: 

# defragment -vn staff_dmn
defragment: Gathering data for 'staff_dmn'
Current domain data:
   Extents:                 263675
   Files w/ extents:        152693
   Avg exts per file w/exts:  1.73
   Aggregate I/O perf:         70%
   Free space fragments:     85574
                <100K   <1M   <10M   >10M
   Free space:   34%   45%    19%     2%
   Fragments:  76197  8930    440      7

You can use the showfile command to check the number of file extents of individual files. For example, to show the attributes and extent information for the orig_file_1 file: 

# showfile -x orig_file_1
    Id Vol PgSz Pages XtntType Segs SegSz   I/O Perf  File
6.8002   2   16    71   simple   **    ** async  82%  orig_file_1
        extentMap: 1
            pageOff    pageCnt    vol    volBlock    blockCnt
                  0          5      2       40720          80
                  5         12      2       41856         192
                 17         16      2       40992         256
                 33          7      2       42048         112
                 40         12      2       41360         192
                 52         15      2       42160         240
                 67          4      2       41792          64
            extentCnt: 7

5.2.2.2    Defragment Example

The following example defragments the accounts_domain file domain from the command line. (See Section 6.4.5.2 for an explanation of running defragment using the GUI.) A time limit of 15 minutes is imposed. Verbose mode is requested to display the fragmentation data at the beginning of each pass through the file domain and at the end of the defragmentation process. 

# defragment -v -t 15 accounts_domain
defragment:  Defragmenting domain 'accounts_domain'
 
Pass 1; 
  Volume 2: area at block      144 (  130800 blocks): 0% full
  Volume 1: area at block   468064 (  539008 blocks): 49% full
  Domain data as of the start of this pass:
    Extents:                   7717
    Files w/extents:           6436
    Avg exts per file w/exts:  1.20
    Aggregate I/O perf:         78%
    Free space fragments:       904
                    <100K    <1M    <10M    >10M
     Free space:      4%     5%     12%     79%
     Fragments:      825     60      13       6
Pass 2;
  Volume 1: area at block   924288 (  547504 blocks): 69% full
  Volume 2: area at block      144 (  130800 blocks):  0% full
  Domain data as of the start of this pass:
    Extents:                   6507
    Files w/extents:           6436
    Avg exts per file w/exts:  1.01
    Aggregate I/O perf:         86%
    Free space fragments:      1752
                    <100K    <1M    <10M    >10M
     Free space:      8%     13%     11%     67%
     Fragments:     1574     157      15       6
 
Pass 3;
  Domain data as of the start of this pass:
    Extents:                   6522
    Files w/extents:           6436
    Avg exts per file w/exts:  1.01
    Aggregate I/O perf:         99%
    Free space fragments:       710
                    <100K    <1M    <10M    >10M
     Free space:      3%    11%     21%     65%
     Fragments:      546    126      32       6
 
Defragment: Defragmented domain 'accounts_domain'

Information displayed before each pass and at the conclusion of the defragmentation process indicates the amount of improvement made to the file domain. A decrease in the Extents and Avg exts per file w/extents values indicates a reduction in file fragmentation. An increase in the Aggregate I/O perf value indicates improvement in the overall efficiency of file-extent allocation.

5.2.3    Balancing a Multivolume File Domain

The balance utility distributes the percentage of used space evenly between volumes in a multivolume file domain created with the optional AdvFS Utilities. This improves performance and evens the distribution of future file allocations.

Figure 5-2:  Balancing a File Domain

Files are moved from one volume to another, as illustrated in Figure 5-2, until the percentage of used space on each volume in the domain is as equal as possible. Because the balance utility does not generally split files, file domains with very large files may not balance as evenly as file domains with smaller files.

To redistribute files across volumes use the AdvFS GUI (see Chapter 6) or, from the command line, enter the balance command:

balance domain_name

This functionality is also available from the SysMan menu.

If you interrupt the balance process, all relocated files remain at their new locations. The rest of the files remain in their original locations.

The following restrictions apply to running the balance utility:

See balance(8) for more information.

5.2.3.1    Choosing to Balance

To determine if you need to balance your files across volumes, use the showfdmn command to display file domain information. From the % used field you can determine if the files are evenly distributed.

Use the balance utility to even file distribution after you have added a volume with the addvol command or removed a volume with the rmvol command (if there are multiple volumes remaining).

In the following example, the usr_domain file domain is not balanced. Volume 1 has 63% used space while volume 2 has 0% used space (it has just been added). 

# showfdmn usr_domain
            Id       Date Created      LogPgs Version Domain Name
3437d34d.000ca710 Mon Jul 5 10:50:05 1999 512       4 usr_domain
 
 Vol  512-Blks   Free % Used  Cmode Rblks  Wblks  Vol Name 
  1L   1488716 549232    63%     on   128    128  /dev/disk/dsk0g
  2     262144 262000     0%     on   128    128  /dev/disk/dsk4a
     --------- -------  ------
       1750860 811232    54%

5.2.3.2    Balance Example

The following example balances the multivolume domain usr_domain, examined above. 

# balance usr_domain
 balance: Balancing domain 'usr_domain' 
 balance: Balanced domain 'usr_domain'
# showfdmn usr_domain
            Id       Date Created      LogPgs Version Domain Name
3437d34d.000ca710 Mon Jul 5 10:50:05 1999 512       4 usr_domain
 
 Vol  512-Blks   Free % Used  Cmode Rblks  Wblks  Vol Name 
  1L   1488716 689152    54%     on   128    128  /dev/disk/dsk0g
  2     262144 122064    53%     on   128    128  /dev/disk/dsk4a
     --------- -------  ------
       1750860 811216    54%

The balance utility moved files from volume 1 to volume 2 in order to even the percentage of used space between the two volumes.

5.2.4    Moving Files to Different Volumes

If you suspect that a fileset or file domain is straining system resources, run the iostat utility (see iostat(1)). If the filesets or file domains are located on devices that appear to be a bottleneck, you can migrate files or pages of files to equalize the load. If a high-performance device is available, you can move an I/O-intensive file to the more efficient volume.

If you do not have AdvFS Utilities, you must go through the dump and restore procedure to move files to a new file domain created on a different device. Because files that are being modified at the time of the copy may not copy correctly, consider mounting the filesets to be moved as read-only or keeping users from accessing the filesets before moving your files.

To move files using the dump and restore procedure:

  1. Make a new file domain on the new device. It must have a temporary new name.

  2. For each fileset in the old file domain, create a fileset with the same name in the corresponding new file domain.

  3. Create a temporary mount point directory.

  4. Mount the new filesets on the temporary mount point.

  5. Use the vdump command to copy the filesets from the old device. Use the vrestore command to restore them to the newly mounted filesets.

  6. Unmount the old and new filesets.

  7. Rename the new file domain to the old name. Since you have not changed the file domain and fileset names, it is not necessary to edit the /etc/fstab file.

  8. Mount the new filesets using the mount points of the old filesets. The directory tree will then be unchanged. Delete the temporary mount point directory.

If you are running Version 5.0 or later, the new file domain is created with the new DVN of 4 (see Section 2.3.3). However, if you must retain the DVN of 3 in order to use earlier versions of the operating system, see mkfdmn(8). The vdump and vrestore utilities are not affected by the change of DVN.

The following example moves the file domain accounts with the fileset technical to volume dsk3c using the same fileset names. The file domain new_accounts is the temporary file domain. Assume the fileset accounts#technical is mounted on /technical. Assume that the /etc/fstab file has an entry to mount accounts#technical on /technical. 

# mkfdmn /dev/disk/dsk3c new_accounts
# mkfset new_accounts technical
# mkdir /tmp_mnt
# mount new_accounts#technical /tmp_mnt
# vdump -dxf - /technical|vrestore -xf - -D /tmp_mnt
# umount /technical
# umount /tmp_mnt
# rmfdmn accounts
# mv /etc/fdmns/new_accounts/ /etc/fdmns/accounts/
# mount /technical
# rmdir /tmp_mnt

If you have the optional AdvFS Utilities, you can use the migrate utility to move heavily accessed or large files to a different volume in the file domain. The balance and defragment utilities also migrate files but are not under user control. With the migrate command, you can specify the volume where a file is to be moved or allow the system to pick the best space in the file domain. You can migrate either an entire file or specific pages to a different volume. Figure 5-3 illustrates the migrate process.

Figure 5-3:  Migrating Files

To move an entire file to a specific volume, use the migrate command with the -d option: 

migrate -s -d destination_vol_index file_name

A file that is migrated will be defragmented in the process if possible. This means that you can use the migrate command to defragment selected files.

The following restrictions apply to the migrate utility:

5.2.4.1    Choosing to Migrate

Choose the migrate utility over the balance utility when you want to control the files that are moved. The balance utility moves files only to optimize distribution. For example, it might move many small files when moving a single larger one would be a better solution for your system.

Choose the migrate utility over the defragment utility when you want to defragment an individual file. If you have a large enough contiguous area on disk, you can migrate the file to that area to defragment it.

You can use the showfile -x command to look at the extent map and the performance percentage of a file. A low performance percentage (less than 80%) indicates that the file is fragmented on the disk. The extent map shows whether the entire file or a portion of the file is fragmented.

The following example displays the extent map of a file called src. The file, which resides in a two-volume file domain, shows an 18% performance efficiency: 

# showfile -x src
    Id Vol PgSz Pages XtntType  Segs  SegSz  I/O  Perf  File
8.8002   1   16    11   simple    **     ** async  18%  src
             extentMap: 1
        pageOff    pageCnt     vol    volBlock    blockCnt
              0          1       1      187296          16
              1          1       1      187328          16
              2          1       1      187264          16
              3          1       1      187184          16
              4          1       1      187216          16
              5          1       1      187312          16
              6          1       1      187280          16
              7          1       1      187248          16
              8          1       1      187344          16
              9          1       1      187200          16
             10          1       1      187232          16
        extentCnt: 11

The file src consists of 11 file extents. This file would be a good candidate to move to another volume to reduce the number of file extents.

5.2.4.2    Migrate Example

The following example migrates the file src examined above. The number of file extents is decreased: 

# migrate -d 2 src
# showfile -x src
    Id Vol PgSz Pages XtntType Segs SegSz  I/O  Perf  File
8.8002   1   16    11   simple   **    ** async 100%  src
   extentMap: 1
      pageOff    pageCnt     vol    volBlock    blockCnt
            0         11       2       45536         176
      extentCnt: 1

The file src now resides on volume 2, consists of one file extent, and has a 100% performance efficiency. Note that in the output above, the first data line of the display lists the metadata. The metadata does not migrate to the new volume. It remains in the original location. The extentMap portion of the display lists the migrated files.

You can tailor the migrate utility to the needs of your system. The following examples illustrate some possibilities.

You can migrate file abc and let the system pick a new location in the file domain:

# migrate abc

You can migrate pages 10 through 99 (that is, 90 pages) of file abc to volume 2 of the file domain:

# migrate -p 10 -n 90 -d 2 abc

You can move the pages of a striped file to different volumes within a file domain. For example, if the abc file is striped across three volumes (volumes 1, 2, 3) of a six-volume file domain, you can use the migrate utility to move the pages from volume 2 to volume 4. You must specify the page offset and the page count for the pages you want to move in addition to the source and destination volume information. Use the showfile command to determine the page count. See migrate(8) for a detailed example.

5.2.5    Striping Files in a File Domain

Striping increases sequential read/write performance by allocating storage in segments across more than one volume. If you have the optional AdvFS Utilities, you can stripe individual files across a number of volumes. For example, if you stripe files with heavy input/output requirements, the I/O is spread across the volumes so that read/write requests to the different disk drives can be overlapped. The stripe utility, in contrast to Logical Storage Manager (LSM) striping, does not require you to stripe all your files.

Figure 5-4:  Creating Striped Files

The stripe utility, as illustrated in Figure 5-4, lets you direct a file to distribute segments across specific disks (or volumes) within a file domain. Zero-length files are striped before data is written to the file. The stripe width is 64 kilobytes. However, you can specify the number of volumes over which your file is striped.

As the file is appended, AdvFS determines the number of pages per stripe segment; the segments alternate among the disks in a sequential pattern. For example, the file system allocates the first segment of a two-disk striped file on the first disk and the next segment on the second disk. This completes one sequence, or stripe. The next stripe starts on the first disk, and so on. Because AdvFS spreads the I/O of the striped file across the specified disks, the sequential read/write performance of the file increases.

To stripe a file and specify the number of volumes in the file domain over which the file should be striped, enter the stripe command:

stripe -n volume_count filename

To stripe a file:

  1. Create a new, empty file.

  2. Stripe the new, empty file specifying the number of volumes over which the file should be striped.

  3. If you are striping an existing file, copy the contents of the old file into the striped file.

For example, to stripe the file shift, which already contains data, across two volumes in the file domain: 

# touch newshift
# stripe -n 2 newshift
# cp shift newshift

You cannot use the stripe utility to modify the number of disks that an already striped file crosses or to restripe a file that is already striped. To change the configuration of a striped file, you must create a new file, stripe it, then copy the original file data to it.

5.2.5.1    Choosing to Stripe

Before you use the stripe utility, run the iostat utility (see iostat(1)) to determine if disk I/O is the bottleneck operation. The blocks per second and transactions per second should be cross checked with the drive's sustained transfer rate. If the disk access is slow, then striping is one of the ways to improve performance (see Section 7.4). Maximum stripe performance will be achievable if each stripe disk is on its own disk controller.

Do not use LSM striping and AdvFS striping at the same time. This has the potential to degrade performance (see Section 2.8.3).

5.2.5.2    Stripe Example

The following example creates an empty file, stripes it, copies data into the striped file, then shows the extents of the striped file:

  1. Create the empty file file_1 and stripe it across three volumes in a domain: 

    # touch file_1
# ls -l file_1
-rw-r--r-- 1 root  system 0 Oct 14 11:06 file_1
# stripe -n 3 file_1

  2. Copy the data from the original file to the striped file: 

    # cp orig_file_1 file_1

  3. Examine the extents of the new striped file: 

    # showfile -x file_1
    Id Vol PgSz Pages XtntType Segs SegSz I/O   Perf File
7.8001   1   16    71   stripe    3     8 async 100% file_1
  extentMap: 1
     pageOff   pageCnt   volIndex  volBlock   blockCnt
           0         8          2     42400        384
          24         8
          48         8
     extentCnt: 1
  extentMap: 2
     pageOff   pageCnt   volIndex   volBlock   blockCnt
           8         8          3      10896        384
          32         8
          56         8
     extentCnt: 1
 
  extentMap: 3
     pageOff   pageCnt   volIndex   volBlock  blockCnt
          16         8          1     186784       368
          40         8
          64         7
     extentCnt: 1

5.2.5.3    Removing Striping

You can alter the pattern of striping in your file domain:

5.2.6    Changing Attributes to Improve System Performance

A number of attributes can be changed to improve system performance. System Configuration and Tuning details the significance of each and the trade-offs engendered when they are changed. See sysconfig(8) for more information. You can modify attributes to:

5.2.7    Controlling Domain Panic Information

The AdvfsDomainPanicLevel attribute allows you to choose whether to have crash dumps created when a domain panic occurs. Values of the attribute are:

See sysconfig(8) for information on changing attributes. See Section 7.5.3 for information about recovering from a domain panic.

5.3    Using a Trashcan

If you have the optional AdvFS Utilities, end users can configure their systems to retain a copy of files they have deleted. Trashcan directories can be attached to one or more directories within the same fileset. Once attached, any file deleted from an attached directory is automatically moved to the trashcan directory. The last version of a file deleted from a directory with a trashcan attached can be returned to the original directory with the mv command.

Root user privilege is not required to use this command. However, the following restrictions apply:

Table 5-1 lists and defines the commands for setting up and managing a trashcan:

Table 5-1:  Trashcan Commands

Command Description
mktrashcan Creates the trashcan.
shtrashcan Shows the contents of the trashcan.
rmtrashcan Removes the trashcan directory.

For example, to attach the trashcan directory keeper to the directory booklist: 

# mkdir keeper
# mktrashcan keeper /booklist
   'keeper' attached to '/booklist'

To remove a file, and look for it in the trashcan directory: 

# rm old_titles
# shtrashcan /booklist
   '//keeper' attached to '/booklist'
#cd /booklist
# ls
   old_titles

To remove the connection between the trashcan and the directory: 

# rmtrashcan /booklist
   '/booklist' detached