AdvFS Guide to File System Administration: Chapter 5

## Chapter 5: Performance Tuning

- - -

AdvFS provides a number of ways to tune your file system's performance. You can configure your filesets and domains to optimize performance. You can run utilities that defragment, balance, stripe, and migrate your files. You can adjust how AdvFS performs logging, caching, and storage allocation.

This chapter includes the following sections:

Configuring for Performance

- Fileset and File Limits
- File-Domain Limits
- Volume Configuration

Utilities for Tuning

- Defragment Utility

- Choosing to Defragment
- Defragment Example

- Balance Utility

- Choosing to Balance
- Balance Example

- Stripe Utility

- Choosing to Stripe
- Stripe Example
- Choosing Between AdvFS and LSM Striping

- Migrate Utility

- Choosing to Migrate
- Migrate Example

Improving the Transaction Log Performance

## Configuring for Performance

-

To ensure optimal performance, AdvFS limits the number of active filesets, file domains, and volumes that it supports. The following sections outline AdvFS file-system limits.

## Fileset and File Limits

- -

While AdvFS allows an unlimited number of filesets per system, only 512 filesets minus the number of active file domains can be mounted simultaneously.

The number of files per fileset is limited to 232, if not first limited by disk space or by quotas.

The currently tested maximum file size is 128 gigabytes. Larger file sizes are allowed, but have not been tested. Sparse files can be addressed to 16 terabytes.

## File-Domain Limits

- -

AdvFS supports a maximum of 100 active file domains per system. A file domain is active when at least one fileset within that domain is mounted. The maximum file-domain size is 128 terabytes.

Each file domain retains its own transaction log. If the file domain has a large number of volumes or includes large numbers of filesets, the log can create a bottleneck because of high activity. See Improving the Transaction Log Performance.

It is important to consider the number of filesets in your domain when setting up your system because each active file domain counts against the mounted-fileset limit. Table 9 shows how the mounted fileset count increases as the number of active file domains increases.

Table 9 Fileset Count

Active Domains | Filesets Mounted | Mounted Fileset Count
1 | 2 | 3
1 | 99 | 100
99 | 99 (1 per domain) | 198

## Volume Configuration

- -

You can add up to 250 volumes to an AdvFS file domain. However, without disk mirroring, it is inadvisable to add more than three volumes. If you lose a volume, the entire domain becomes inaccessible. The risk of losing a volume, and thus losing access to your file domain, increases as the number of volumes increases.

There is a small performance advantage to dividing disks on different SCSI chains. However, if you must purchase additional controllers, this can be an expensive way to provide a minimal performance improvement in data access.

The theoretical maximum AdvFS volume size is 1 terabyte.

## Utilities for Tuning

- -

AdvFS provides utilities for keeping your file system running as efficiently as possible. These tools improve read/write performance by altering the way files are mapped on the disk. They can be run while the system is on line and are transparent to system users and to applications.

You can improve file read/write performance on your system with utilities that:

- Defragment a file domain.
- Balance a multivolume file domain to even the storage distribution between volumes.
- Stripe files across volumes in a multivolume file domain.
- Migrate files to a different volume within a multivolume file domain.

## Defragment Utility

- -

AdvFS attempts to store file data in contiguous blocks on a disk. This collection of contiguous blocks is called a file extent. If all data in a file is stored in contiguous blocks, that file has one file extent. However, as files grow, contiguous blocks on the disk may not be available to accommodate the new data. The system must spread the file over discontiguous blocks. As a result, the file is fragmented on the disk and consists of multiple file extents. File fragmentation degrades the read/write performance because many disk addresses must be examined to access a file.

Figure 7 Defragment File Domain

The `defragment` utility reduces the amount of file fragmentation in a file domain by attempting to make the files more contiguous so that the number of file extents is reduced. Defragmentation is an iterative, two-step process that operates on the file domain, as follows:

1. Files are moved out of a region to create an area with contiguous, unallocated space.
2. Fragmented files are written into a region that has more contiguous space so they are less fragmented.

In addition to making files contiguous so that the number of file extents is reduced, defragmenting a file domain often makes the free space on a disk more contiguous so files that are created later will also be less fragmented.

You can improve the efficiency of the defragment process by deleting any unneeded files in the file domain before running the `defragment` utility.

The following restrictions apply to running the `defragment` utility:

1. You must have root user privileges to access the `defragment` command.
2. All filesets in the file domain must be mounted. If you try to defragment an active file domain that includes unmounted filesets, you will get an error message.
3. A minimum free space amount of 1% of the total space or 5 megabytes per volume (whichever is less) must exist in order to run.
4. The `defragment` utility cannot be run while the `addvol`, `rmvol`, `balance`, or `rmfset` command is running in the same file domain.
5. Striped files are not defragmented.

## Choosing to Defragment

- -

To determine the amount of file fragmentation that exists in a file domain, use the `defragment` command with the `-v` and `-n` flags. This will show how fragmented the file domain is without starting the `defragment` utility.

Table 10 describes the information displayed by the `defragment` utility when you run the command with the verbose (`-v`) flag.

Table 10 Defragment Utility Output

Heading | Description
`Extents` | Number of extents in the specified domain
`Files w/ extents` | Number of files that have extents
`Avg exts per file w/ exts` | Average number of extents for each file that has one or more extents
`Aggregate I/O perf` | Efficiency of the entire file domain
`Free space fragments` | Number of free space fragments in the domain

You can also use the `showfile` command to check the number of file extents of individual files. The following example shows the attributes and extent information for the `mail` file:

    # showfile -x mail
           Id Vol PgSz Pages XtntType Segs SegSz Log Perf  File

    4198.800d   2   16    27   simple   **    ** off  66%  tutorial

            extentMap: 1

                pageOff    pageCnt    vol    volBlock    blockCnt

                      0          5      2      781552          80

                      5         12      2      785776         192

                     17         10      2      786800         160

                   extentCnt: 3

## Defragment Example

- -

The following example defragments the `accounts_domain` file domain. A time limit of 15 minutes is imposed. Verbose mode is requested to display the fragmentation data at the beginning of each pass made through the file domain and at the end of the defragmentation process.

    # defragment -v -t 15 accounts_domain

    defragment:  Defragmenting domain 'accounts_domain'

    Pass 1; Clearing

      Volume 1: area at block    11680 (  103072 blocks): 81% full

      Domain data as of the start of this pass:

        Extents:                  10432

        Files w/extents:           4305

        Avg exts per file w/exts:  2.42

        Aggregate I/O perf:         52%

        Free space fragments:      2743

                        <100K    <1M    <10M    >10M

         Free space:     38%      0%      0%     62%

         Fragments:     2742       0       0       1

    Filling

    .

    .

    .

    Pass 13; Clearing

      Volume 1: area at block   559744 (   62736 blocks):  0% full

      Volume 2: area at block    76640 (   24624 blocks): 18% full

      Domain data as of the start of this pass:

        Extents:                   4306

        Files w/extents:           4305

        Avg exts per file w/exts:  1.00

        Aggregate I/O perf:        100%

        Free space fragments:        23

                        <100K    <1M    <10M    >10M

         Free space:      0%      9%     27%     64%

         Fragments:        6      10       5       2

    Filling

    Current domain data:

       Extents:                   4305

       Files w/extents:           4305

       Avg exts per file w/exts:  1.00

       Aggregate I/O perf:        100%

       Free space fragments:        17

                       <100K    <1M    <10M    >10M

        Free space:      0%      6%     29%     65%

        Fragments:        3       8       4       2

    Defragment: Defragmented domain 'accounts_domain'

Information displayed before each pass and at the conclusion of the defragmentation process indicates the amount of improvement made to the file domain. A decrease in the `Extents` and `Avg exts per file w/extents` values indicates a reduction in file fragmentation. An increase in the `Aggregate I/O perf` value indicates improvement in the overall efficiency of file-extent allocation.

## Balance Utility

- -

The `balance` utility distributes the percentage of used space evenly between volumes in a multivolume file domain. This improves performance and evens the distribution of future file allocations.

Figure 8 Balance File Domain

Files are moved from one volume to another until the percentage of used space on each volume in the domain is as equal as possible. (Because the `balance` utility does not generally split files, file domains with very large files may not balance as evenly as file domains with smaller files.)

The following restrictions apply to running the `balance` utility:

1. You must have root user privileges to access the `balance` command.
2. All filesets in the file domain must be mounted. If you try to balance an active file domain that includes unmounted filesets, you will get an error message.
3. A minimum free space amount of 1% of the total space or 5 megabytes per volume (whichever is less) must exist in order to run.
4. The `balance` utility cannot run while the `addvol`, `rmvol`, `defragment`, or `rmfset` command is running in the same file domain.

## Choosing to Balance

- -

To determine if you need to balance your files across volumes, use the `showfdmn` command to display file-domain information. From the `% used` field you can determine if the files are evenly distributed.

In the following example, the `usr_domain` file domain is not balanced. Volume 1 has 85% used space while volume 2 has 0% used space (it has just been added).

    # showfdmn usr_domain

                   Id      Date Created         LogPgs  Domain Name

    2dcab512.000d1ed0  Fri May  6 14:22:26 1994    512  usr_domain

      Vol   512-Blks    Free  % Used  Cmode  Rblks  Wblks  Vol Name

       1L     819200  126848     85%     on    128    128 /dev/rz8g

       2      768281  768080      0%     on    128    128 /dev/rz9d

          ----------  -------  ------

             1587481  894928     44%

Use the `balance` utility to even file distribution after you have added a volume with the `addvol` command or removed a volume with the `rmvol` command (if there are multiple volumes remaining).

## Balance Example

- -

The following example balances the multivolume domain `usr_domain`, examined above:

    # balance usr_domain

     balance: Balancing domain 'usr_domain'

     balance: Balanced domain 'usr_domain'

    # showfdmn usr_domain

                   Id      Date Created         LogPgs  Domain Name

    2dcab512.000d1ed0  Fri May  6 14:22:26 1994    512  usr_domain

      Vol   512-Blks    Free  % Used  Cmode  Rblks  Wblks  Vol Name

       1L     819200  459248     44%     on    128    128 /dev/rz8g

       2      768281  433632     44%     on    128    128 /dev/rz9d

          ----------  -------  ------

             1587481  892880     44%

The `balance` utility moved files from volume 1 to volume 2 in order to even the percentage of used space between the two volumes.

## Stripe Utility

- -

Striping increases sequential read/write performance by allocating storage in segments across more than one volume. AdvFS allows you to stripe individual files according to your needs. For example, if you stripe files that have heavy input/output requirements, the I/O is spread across the volumes, so that read/write requests to the different disk drives can be overlapped. The AdvFS striping utility does not require you to stripe all your files.

With the `stripe` utility, you direct a file to distribute segments across specific volumes within a file domain. You first create a new zero-length file, stripe it, then write your data to the striped file and delete the original file. You can choose the number of volumes on which to stripe a file.

Figure 9 Stripe Files

As the file is appended, AdvFS determines the number of pages per stripe segment; the segments alternate among the disks in a sequential pattern. For example, the file system allocates the first segment of a two-disk striped file on the first disk and the next segment on the second disk. This completes one sequence, or stripe. The next stripe starts on the first disk, and so on.

You cannot use the `stripe` utility to modify the number of disks that an already striped file crosses or to restripe a file that is already striped. To change the configuration of a striped file, you must repeat the striping process.

## Choosing to Stripe

- -

Before you use the `stripe` utility, run the `iostat` utility to determine if disk I/O is the bottleneck operation. The blocks per second and transactions per second should be cross checked with the drive's sustained transfer rate. If the disk access is slow, then striping will improve performance. See Checking Disk Activity.

## Stripe Example

- -

The following example creates an empty file, stripes it, copies data into the striped file, then shows the extents of the striped file:

1. Create the empty file`file_1` and stripe it across three volumes in a domain:

       # touch file_1

       # ls -l file_1

       -rw-r--r-- 1 root system 0 Aug 1 05:50 file_1

       # stripe -n 3 file_1

2. Copy the data from the original file to the striped file:

       # cp orig_file_1 file_1

3. Examine the extents of the new striped file:

       # showfile -x file_1

       Id            Vol PgSz Pages XtntType  Segs  SegSz   Log   Perf   File

       1ed.8053        1   16    115  stripe    3       8    off   75%   file_1

         extentMap: 1

            pageOff   pageCnt   volIndex  volBlock   blockCnt

                  0         8          2       160        384

                 24         8

                 48         8

            extentCnt: 1

          extentMap: 2

            pageOff   pageCnt   volIndex   volBlock   blockCnt

                  8         8          3        128        336

                 32         8

                 56         5

            extentCnt: 1

         extentMap: 3

            pageOff   pageCnt   volIndex   volBlock  blockCnt

                 16         8          1       8528       256

                 40         8

            extentCnt: 1

          extentCnt: 1

## Choosing Between AdvFS and LSM Striping

- -

AdvFS implements file striping at an individual file level. Therefore, if only one or a few files are large and have heavy I/O requirements, use the AdvFS `stripe` utility to stripe these files.The Logical Storage Manager (LSM) provides volume-level striping. With LSM you preconfigure your volumes for file striping. Then, all files located on the volumes configured for striping will be striped.

## Migrate Utility

- -

The `migrate` utility improves file I/O performance by altering the way files are mapped on the disk. You can use the `migrate` utility to move heavily accessed or large files to a different volume in the file domain. You can specify the volume where a file is to be moved or allow the system to pick the best space in the file domain. You can migrate either an entire file or specific pages to a different volume.

Figure 10 Migrate Files

A file that is migrated will be defragmented in the process if possible. This means that you can use the `migrate` command to defragment selected files.

The following restrictions apply to the `migrate` utility:

- You must have root-user privilege to access this utility.
- You can only perform one migrate operation at a time on the same file.
- When you migrate a striped file, you can only migrate one source volume at a time.
- The `migrate` utility does not evaluate your migration decisions. For example, you can move more than one striped file segment to the same disk, which defeats the purpose of striping the file.

## Choosing to Migrate

- -

If you suspect that a fileset or file domain is straining system resources, run the `iostat` utility. If the filesets or file domains are located on devices that appear to be a bottleneck, you can migrate files or pages of files to equalize the load. If a high-performance device is available, you can move an I/O-intensive fileset to the more efficient volume. See Checking Disk Activity.

Choose the `migrate` utility over the `balance` utility when you want to control the files that are moved. The `balance` utility moves files only to optimize distribution. For example, it might move many small files when moving a single larger one would be a better solution for your system.

You can use the `showfile` command with the `-x` flag to check the performance percentage and extent map of a file. A low performance percentage (less than 80%) indicates that the file is fragmented on the disk. The extent map shows whether the entire file or a portion of the file is fragmented.

The following example displays the extent map of a file called `src`. The file, which resides in a two-volume file domain, has an 18% performance efficiency. The `showfile` command with the `-x` flag lists the extent map for the file `src`:

`#` showfile -x src

           Id Vol PgSz Pages XtntType  Segs  SegSz  Log  Perf  File
    2e1b.8004   1   16    11   simple    **     **  off   18%  src
        extentMap: 1
            pageOff    pageCnt     vol    volBlock    blockCnt
                  0          1       1      684704          16
                  1          1       1      684896          16
                  2          1       1      684928          16
                  3          1       1      685280          16
                  4          1       1      685312          16
                  5          1       1      685488          16
                  6          1       1      686432          16
                  7          1       1      686608          16
                  8          1       1      686784          16
                  9          1       1      686960          16
                 10          1       1      687168          16
            extentCnt: 11

The file `src` consists of 11 file extents. This file would be a good candidate to move to another volume to reduce the number of file extents.

## Migrate Example

- -

The following example migrates the file `src` examined above. The number of file extents is decreased:

    # migrate -d 2 src

    # showfile -x src

           Id Vol PgSz Pages XtntType Segs SegSz  Log  Perf  File
    2e1b.8004   1   16    11   simple   **    **  off  100%  src
        extentMap: 1
                pageOff    pageCnt     vol    volBlock    blockCnt
                      0         11       2        1280         176
                extentCnt: 1

The file `src` now resides on volume 2, consists of one file extent, and has a 100% performance efficiency. Note that in the output above, the first data line of the display lists the metadata. The metadata does not migrate to the new volume. It remains in the original location. The `extentMap` portion of the display lists the migrated files.

You can tailor the `migrate` utility to the needs of your system. The following examples illustrate some possibilities.

You can migrate file `abc` and let the system pick a new location in the file domain:

    # migrate abc

You can migrate pages 10 through 99 of file `abc` to volume 2 of the file domain:

    # migrate -p 10 -n 90 -d 2 abc

You can move the pages of a striped file to different volumes within a file domain. For example, if the `abc` file is striped across three volumes (volumes 1, 2, 3) of a 6-volume file domain, you can use the `migrate` utility to move the pages from volume 2 to volume 4. You must specify the page offset and the page count for the pages you want to move in addition to the source volume information. Use the `showfile` command to determine the page count. In the following example, volume 2 contains pages between 8 and 39 of the striped file. To migrate the pages, enter:

    # migrate -p 8 -n 32 -s 2 -d 4 abc

The `abc` file is now striped across volumes 1, 3, and 4.

## Improving the Transaction Log Performance

-

Each file domain has a transaction log that keeps track of fileset activity for all filesets in the file domain. If the log resides on a congested disk or bus, or the file domain contains many filesets, system performance can degrade.

Use the `showfdmn` command to determine the location of the log. The letter L after the volume number indicates the volume on which the log resides. You can monitor performance of this volume with the `iostat` utility. If the volume containing the log appears overloaded, you can:

- Divide the file domain into several smaller domains. Then, each transaction log will handle transactions for fewer filesets.
- Use the `switchlog` command to move the log to a faster or less congested volume.

The `switchlog` command allows you to move the transaction log to any volume in the domain. Refer to the `switchlog`(8) reference page for more information. Moving the log to a faster device can improve performance. Issue the `switchlog` command with the following format:

    switchlog domain_name new_volume_number

Moving the transaction log may also be useful when you are using LSM and wish to increase reliability by placing your transaction log on a volume that is mirrored.

- - - -

© Copyright Digital Equipment Corporation 1996
All rights reserved