10    Using Logical Storage Manager in a Cluster

This chapter presents configuration and usage information that is specific to Logical Storage Manager (LSM) in a TruCluster Server environment. The chapter discusses the following subjects:

• Understanding differences between managing LSM in clusters and standalone systems (Section 10.1)

• Understanding storage connectivity and LSM volumes (Section 10.2)

• Configuring LSM for a cluster (Section 10.3)

• Adding cluster members with LSM legacy volumes (Section 10.4)

• Moving LSM disk groups between standalone systems and clusters (Section 10.5)

• Configuring dirty-region log sizes (Section 10.6)

• Encapsulating the /usr file system, the cluster root domain, other Advanced File System (AdvFS) domains, or an individual member's swap devices into LSM volumes (Section 10.7)

For complete documentation on LSM, see the Tru64 UNIX Logical Storage Manager manual. Information on installing LSM software can be found in that manual and the Tru64 UNIX Installation Guide.

Using LSM in a cluster is like using LSM on a single system. The same LSM software subsets are used for both clusters and standalone configurations.

In a cluster, LSM provides the following features:

• High availability

  LSM operations continue despite the loss of cluster members, as long as the cluster itself continues operation and a physical path to the storage is available.

• Performance:

  • For I/O within the cluster environment, LSM volumes incur no additional LSM I/O overhead.

    LSM follows a fully symmetric, shared I/O model, where all members share a common LSM configuration and each member has private dirty-region logging.

  • Disk groups can be used simultaneously by all cluster members.

  • There is one shared rootdg disk group.

  • Any member can handle all LSM I/O directly, and does not have to pass it to another cluster member for handling.

• Ease of management

  The LSM configuration can be managed from any member.

10.1    Differences Between Managing LSM in Clusters and in Standalone Systems

The following restrictions apply to LSM in a cluster:

• LSM volumes cannot be used for the boot partitions of individual members.

• LSM cannot be used to mirror a quorum disk or any partitions on that disk.

• LSM Redundant Array of Independent Disks (RAID) 5 volumes are not supported in clusters.

• To place the cluster_root domain under LSM control, you must use the volmigrate command. To place the cluster_usr or cluster_var domains under LSM control, use either the volmigrate command or the volencap command.

• There are small but important differences in the process of configuring LSM. See Section 10.3.

• The size requirements for log subdisks in a cluster differ from those in a standalone system. See Section 10.6.

The following LSM behavior in a cluster varies from the single-system image model:

• Statistics that are returned by the volstat command apply only to the member on which the command executes.

• The voldisk list command can give different results on different members for disks that are not part of LSM (that is, autoconfig disks).

  The differences are typically limited to disabled disk groups. For example, one member might show a disabled disk group and on another member that same disk group might not show at all.

10.2    Storage Connectivity and LSM Volumes

When adding disks to an LSM disk group on a cluster, note the following points:

• Make sure that all storage in an LSM volume has the same connectivity. LSM volumes have the same connectivity when either one of the following conditions is true:

  • All disks in an LSM disk group are on the same shared SCSI bus.

  • Disks in an LSM disk group are on different shared SCSI buses, but all of those buses are connected to the same cluster members.

• Storage availability increases as more members have direct access to all disks in a disk group.

  Availability is highest when all disks in a disk group are on a shared bus that is directly connected to all cluster members.

• Private disk groups (a disk group whose volumes are all connected to the private bus of a single cluster member) are supported, but if that member becomes unavailable, then the cluster loses access to the disk group.

  Because of this, a private disk group is suitable only when the member that the disk group is physically connected to is also the only member that needs access to the disk group.

• Avoid configuring a disk group with volumes that are distributed among the private buses of multiple members. Such disk groups are not recommended, because no single member has direct access to all volumes in the group.

The drdmgr and hwmgr commands can give you information about which systems serve which disks. To get a graphical display of the cluster hardware configuration, including active members, buses, storage devices, and their connections, use the sms command to invoke the graphical interface for the SysMan Station, and then select Hardware from the Views menu.

10.3    Configuring LSM for a Cluster

The procedure for configuring LSM for a cluster depends on the state of the system where LSM is to be configured. There are three possibilities:

• Tru64 UNIX has been installed on the system, but the system has not yet been initiated as a cluster (you have not run the clu_create command). See Section 10.3.1.

• The initial cluster member has been created (you have run the clu_create command), but it is still a single-member cluster (you have not yet run the clu_add_member command). See Section 10.3.2.

• The cluster has been created and members have been added, forming a multimember cluster. See Section 10.3.3.

10.3.1    Configuring LSM Before Cluster Creation

If you configured LSM on the Tru64 UNIX system before cluster creation, the existing LSM configuration is propagated to the cluster when the cluster is created.

All LSM volumes from the system are available on the new cluster, including volumes created by encapsulating the single system's boot partitions and swap space. However, the original system-related volumes are not used after the system is converted to a cluster, and the domains are not available until explicitly mounted. If you halt the cluster and boot the base operating system again, these domains are still available.

When you add new members to the cluster, they are automatically configured to run LSM. You do not need to do any further LSM configuration.

For information about installing LSM software and configuring LSM on a Tru64 UNIX system before cluster creation, see the Tru64 UNIX Logical Storage Manager manual.

10.3.2    Configuring LSM After Cluster Creation and Before Members Have Been Added

The procedure for configuring LSM after cluster creation and before any members have been added is the same as the procedure for configuring LSM on a standalone Tru64 UNIX system. See the chapter on setting up the LSM software in the Tru64 UNIX Logical Storage Manager manual.

After LSM is configured on the initial cluster member, LSM is automatically configured on the new members as they are added to the cluster. You do not need to do any further LSM configuration.

10.3.3    Configuring LSM in a Multimember Cluster

To configure LSM on an established multimember cluster, follow these steps after installing the LSM software:

1. Enter the following command on one cluster member. It does not matter which member.

   # volsetup
   

   You are queried to list disk names or partitions to be added to the rootdg disk group. For more information, see volsetup(8).

2. Synchronize LSM throughout the cluster by running the following command on each of the other cluster members:

   # volsetup -s
   

   Note

   Do not run volsetup -s on the cluster member where you first configured LSM.

   If a new member is later added to the cluster, do not run the volsetup -s command on the new member. The clu_add_member command automatically synchronizes LSM on the new member.

10.4    Adding Cluster Members with LSM Legacy Volumes

If you have a standalone system with LSM volumes, you can make that system a cluster member and incorporate its LSM volumes in the established cluster.

If you want the cluster to be able to use the disks in rootdg, you must rename rootdg as you import it on the cluster. Any volumes in the old rootdg disk group that were not used by the standalone system's root file system, /usr, /var, or swap partitions will be usable on the cluster.

To prepare a standalone system to become a cluster member:

1. Before halting the standalone system in preparation to connecting it to the cluster, deport each disk group (except rootdg) with the voldg command:

   # voldg deport diskgroup
   

2. Display the disks in the rootdg disk group:

   # voldisk -g rootdg list
   

   Information similar to the following is displayed:

   DEVICE       TYPE      DISK         GROUP        STATUS
   dsk1         sliced    dsk1         rootdg       online
   dsk2         sliced    dsk2         rootdg       online
   dsk3         sliced    dsk3         rootdg       online
   dsk4         sliced    dsk4         rootdg       online
   dsk5         sliced    dsk5         rootdg       online
   dsk6         sliced    dsk6         rootdg       online
   dsk7         sliced    dsk7         rootdg       online
   dsk8         sliced    dsk8         rootdg       online
   dsk9         sliced    dsk9         rootdg       online
   
   .
   .
   .
   
   

3. Use the name of any disk in rootdg to identify the disk group ID (DGID) of rootdg:

   # voldisk list dsk1
   

   Information similar to the following is displayed:

   Device:    dsk1
   devicetag: dsk1
   type:      sliced
   hostid:    lsmtemp.xyz.abc.com
   disk:      name=dsk1 id=962390779.1466.lsmtemp.xyz.abc.com
   group:     name=rootdg id=959293418.1026.lsmtemp.xyz.abc.com
   flags:     online ready autoimport imported
   pubpaths:  block=/dev/disk/dsk1g char=/dev/rdisk/dsk1g
   privpaths: block=/dev/disk/dsk1h char=/dev/rdisk/dsk1h
   version:   2.1
   iosize:    min=512 (bytes) max=2048 (blocks)
   public:    slice=6 offset=16 len=8375968
   private:   slice=7 offset=0 len=4096
   update:    time=973707788 seqno=0.35
   headers:   0 248
   configs:   count=1 len=2993
   logs:      count=1 len=453
   Defined regions:
    config   priv     17-   247[   231]: copy=01 offset=000000 enabled
    config   priv    249-  3010[  2762]: copy=01 offset=000231 enabled
    log      priv   3011-  3463[   453]: copy=01 offset=000000 enabled
    
   

   The DGID appears in the line that begins with group:. In the previous output, the DGID of rootdg is 959293418.1026.lsmtemp.xyz.abc.com

4. Add the system to the cluster.

5. After the former standalone system has been added to the cluster, import the disk groups as described in Section 10.5.

10.5    Moving LSM Disk Groups Between Standalone and Cluster Environments

In a cluster, the LSM configuration database of each disk group is marked to indicate that it is being used in a cluster environment. A nonautoimported disk group that is designated for use in a cluster environment cannot be used in a standalone environment. Similarly, a nonautoimported disk group designated for use in a standalone environment cannot be used in a cluster.

Note

To move an LSM disk group between standalone and cluster environments, one of the following must be true:

• The disk group was deported with the voldg command.

• The system or cluster using the disk group was cleanly shut down.

A disk group that requires recovery, for example, due to a system or cluster crash, cannot be moved between standalone and cluster environments. The disk group must first be recovered in the environment where it was last used.

10.5.1    Importing Tru64 UNIX Version 5.1A Standalone Disk Groups

Manually importing a Tru64 UNIX Version 5.1A disk group from a standalone system into a cluster environment requires that you explicitly change the designation of that disk group for use in a cluster. If you move the disk group to a cluster and then boot the cluster, the disk group is imported and its configuration database is converted automatically.

To manually import a disk group from a standalone system and make the volumes available to the cluster, follow these steps:

1. Import the disk group and mark it for use in a cluster by doing one of the following:

   • For all disk groups other than the standalone system's rootdg disk group, enter:

     # voldg -o shared import diskgroup
     

   • To import and convert the standalone system's rootdg disk group, assign it a new name and import it using its DGID as identified in Section 10.4:

     # voldg -o shared -n newname import id=rootdg_dgid
     

2. Start the volumes in the disk group:

   # volrecover -g diskgroup
   

You can also move a disk group from a cluster back to a standalone system. This requires marking the disk group for single system use.

To move a disk group from a cluster environment to a standalone environment:

1. Physically move the disk group to the standalone system.

2. Import the disk group, marking it for single system use:

   # voldg -o private import diskgroup
   

3. Start the volumes in the disk group:

   # volrecover -g diskgroup
   

10.5.2    Importing Tru64 UNIX Version 4.0 Standalone Disk Groups

To import a Tru64 UNIX Version 4.0 disk group from a standalone system to a TruCluster Server environment, you must do the following:

1. Determine the device name, media name, and LSM disk type of each disk.

   You need to identify each of the old rz disks or partitions and their corresponding dsk names. The old rz names identify the device names either directly or by the error entries that are returned by the voldisk list command.

2. Convert the disk group and mark it for use in a cluster.

3. Convert the legacy device special file names.

The rest of this section describes each step in more detail.

10.5.2.1    Determining the Device Name, Media Name, and LSM Disk Types

Determine the access names, media names, and LSM disk types with the following command:

# voldisk list

Use the voldisk list command to determine the old device names that are in an error state, as well as to see what new names were autodiscovered. In the case of multiple nopriv disks, voldisk list returns nothing to differentiate the disks. In this case, you might need to use the hwmgr command to determine physical addresses. Then you have to map these physical addresses to previously recorded information for access names, device names, and media names.

10.5.2.2    Converting the Disk Group for Cluster Use

Note

After a disk group is converted, it cannot be used again on a Tru64 UNIX Version 4.0 system.

To convert the disk group and mark it for cluster use, enter the following command:

# voldg -o convert_old -o shared import diskgroup

10.5.2.3    Converting Legacy Device Special Files

LSM automatic configuration locates all disks and imports them. LSM marks devices with legacy device names as failed.

To convert the legacy device special file names, follow these steps:

1. Remove each device that is marked failed with the following command. Substitute the actual legacy device name for rzNN:

   # voldisk rm rzNN
   

2. Reattach each disk of type nopriv to associate a media name with a device name. For each disk of type nopriv, enter the following command, using the information that you collected in step 1:

   # voldg -k adddisk media_name=device_name
   

10.6    Dirty-Region Log Sizes for Clusters

LSM uses log subdisks to store the dirty-region logs of volumes that have Dirty Region Logging (DRL) enabled. By default, the volassist command configures a log subdisk large enough so that the associated mirrored volume can be used in either a cluster or a standalone system.

For performance reasons, standalone systems might be configured with values other than the default. If a standalone system has log subdisks that are configured for optimum performance, and that system is to become part of a cluster, the log subdisks must be reconfigured with 65 or more blocks.

To reconfigure the log subdisk, use the volplex command to delete the old DRL, and then use volassist to create a new log. You can do this while the volume is active; that is, while users are performing I/O to the volume.

In the following example, the volprint command gets the name of the current log for vol1. Then the volplex command dissociates and removes the old log. Finally, the volassist command creates a new log subdisk for vol1. By default, the volassist command creates a log subdisk of a size that is appropriate to a cluster environment.

# volprint vol1 | grep LOGONLY
pl vol1-03    vol1     ENABLED  LOGONLY    -    ACTIVE    -     -
# volplex -o rm dis vol1-03
# volassist addlog vol1

Note

In a cluster, LSM DRL sizes must be at least 65 blocks for the DRL to be used with a mirrored volume.

If the DRL size for a mirrored volume is less than 65 blocks, DRL is disabled. However, the mirrored volume can still be used.

Table 10-1 lists some suggested DRL sizes for small, medium, and large storage configurations in a cluster. The volassist addlog command creates a DRL of the appropriate size.

Table 10-1:  Sizes of DRL Log Subdisks

10.7    Placing Cluster Domains into LSM Volumes

You can place cluster domains or cluster members' swap devices into LSM volumes. This allows you to use the features of LSM, such as mirroring, on those volumes. The following methods are available on a cluster to place domains and members' swap devices under LSM control:

• Use the volencap and volreconfig commands to encapsulate a domain or swap device into an LSM volume.

  The volencap command creates LSM volumes on the same disks or disk partitions that the domain or swap space is currently using. Use the volencap command to encapsulate the /usr file system, any AdvFS domain other than cluster_root, or the swap devices for a cluster member.

  The advantage to using this method is that no extra disk space is required. The disadvantage is that you might need to shut down and reboot the cluster or cluster member for the encapsulation to complete.

  Section 10.7.1 describes how to encapsulate the /usr file system on a cluster. For information on encapsulating other file systems, domains, or existing data, see the Tru64 UNIX Logical Storage Manager manual and the volencap(8) reference page.

• Use the volmigrate command to migrate an AdvFS domain to an LSM volume.

  The volmigrate command creates LSM volumes on the disks that you specify, moves the domain to the volumes, and removes the original disk from the domain and leaves it unused. The advantage to using this method is that the migration occurs while the domain's filesets are mounted, and no reboot is required. The disadvantage is that the migration process temporarily uses additional disk space while the domain data is copied to the LSM volume.

  To place the cluster_root domain under LSM control, you must use the volmigrate command.

The following sections describe each method in more detail.

10.7.1    Encapsulating the /usr File System

The volencap command creates scripts to perform in-place encapsulation of disk partitions into LSM volumes.

Note

When encapsulating the cluster /usr file system, you must shut down the entire cluster. The scripts that are created by the volencap command are executed during the startup routine.

To encapsulate the /usr file system:

1. Enter the following command:

   # volencap cluster_usr
   

2. Shut down the cluster:

   # shutdown -chs time
   

3. Boot the member to multi-user mode.

4. Boot the remaining cluster members.

After you shut down the cluster, the volreconfig command is automatically executed during system boot from /etc/inittab.

10.7.2    Encapsulating Members' swap Devices

You can encapsulate:

• All the swap devices for a member at once, with the swap operand

• The above plus swap devices for other members, in the same command

• Only the swap devices you specify, for one or more members, in the same command

Note

All members whose swap devices you want to encapsulate must be running.

You can run the volencap command on one member to set up the encapsulation for several members at once. However, you must run the volreconfig command on each member whose swap devices you are encapsulating. The volreconfig command executes the scripts created by the volencap command and reboots the cluster member to complete the encapsulation.

The swap operand is member-specific; it is a shortcut for specifying all the swap devices for the member it is run on. To encapsulate swap devices for several members at a time, you must identify the other members' swap devices, for example by entering the swapon command on each member.

• To encapsulate all the swap devices for one member only, enter:

  # volencap swap
  # volreconfig
  

  After the member reboots, each swap device uses an LSM volume.

• To encapsulate all the swap devices for one member, plus the devices for one or more other members:

  1. Identify the cluster members:

     # clu_get_info
     

  2. Display the swap devices for each member by using one of the following methods:

     • On each member, enter:

       # swapon -s
       

     • On one member, display the swap devices for all members by entering the following command, where n is the member number:

       #  more /cluster/members/membern/boot_partition/etc/sysconfigtab | grep swap
       

  3. Do one of the following:

     • To encapsulate all swap devices for the current member only, enter:

       # volencap swap
       

     • To encapsulate all swap devices for the current member, plus devices for additional members, enter:

       # volencap swap dsknp dsknp ...
       

     • To encapsulate specific swap devices for the current member, plus (optionally) additional members, enter:

       # volencap dsknp dsknp ...
       

  4. Enter the following command on each member with queued-up encapsulation scripts:

     # volreconfig
     

  After each member reboots, its swap devices use LSM volumes.

See the volencap(8) reference page for more information on encapsulating swap in a cluster.

You can create an LSM volume for secondary swap space instead of encapsulating the member's swap devices, or in addition to doing so. See the Tru64 UNIX Logical Storage Manager manual for more information on creating volumes for secondary swap.

10.7.3    Migrating AdvFS Domains into LSM Volumes

You can place an AdvFS domain, including the cluster root domain cluster_root, into an LSM volume. This operation uses a different disk than the disk on which the domain originally resides, and therefore does not require a reboot. You cannot place the individual members' boot partitions (called rootmemberID_domain#root) into LSM volumes.

You can specify:

• The name of the volume (default is the name of the domain with the suffix vol)

• The number of stripes and mirrors that you want the volume to use

  Striping improves read performance, and mirroring ensures data availability in the event of a disk failure.

You must specify LSM disks by their disk media names on which to create the volume for the domain, and all the disks must belong to the same disk group. For the cluster_root domain, the disks must be simple or sliced disks (must have an LSM private region) and must belong to the rootdg disk group. The command fails if you specify disk media names that belong to a disk group other than rootdg.

There must be sufficient LSM disks and they must be large enough to contain the domain. See the volmigrate(8) reference page for more information on disk requirements and the options for striping and mirroring.

To migrate a domain into an LSM volume, enter:

# volmigrate [-g diskgroup] [-m num_mirrors] [-s num_stripes] domain_name disk_media_name...

The volmigrate command creates a volume with the specified characteristics, moves the data from the domain into the volume, removes the original disk or disks from the domain, and leaves those disks unused. The volume is started and ready for use, and no reboot is required.

You can use LSM commands to manage the domain volume the same as for any other LSM volume.

If a disk in the volume fails, see the Troubleshooting section in the Logical Storage Manager manual for the procedure to replace a failed disk and recover the volumes on that disk. If a disk failure occurs in the cluster_root domain volume and the procedure does not solve the problem (specifically, if all members have attempted to boot, yet the volume that is associated with cluster root cannot be started), then you might have to restore the cluster root file system using a backup tape. After restoring the cluster, you can again migrate the cluster root domain to an LSM volume as described here.

10.7.4    Migrating Domains from LSM Volumes to Physical Storage

You can migrate any AdvFS domain onto physical disk storage and remove the LSM volume with the volunmigrate command. The cluster remains running during this process and no reboot is required.

You must specify one or more disk partitions that are not under LSM control, ideally on a shared bus, for the domain to use after the migration. These partitions must be large enough to accommodate the domain plus at least 10 percent additional space for file system overhead. The volunmigrate command examines the partitions that you specify to ensure they meet both qualifications, and returns an error if either or both is not met. See the volunmigrate(8) reference page for more information.

To migrate an AdvFS domain from an LSM volume to physical storage:

1. Determine the size of the domain volume:

   # volprint -vt domainvol
   

2. Find one or more disk partitions on a shared bus that are not under LSM control and are large enough to accommodate the domain plus file system overhead of at least 10 percent:

   # hwmgr -view devices -cluster
   

3. Migrate the domain, specifying the target disk partitions:

   # volunmigrate domain_name dsknp [dsknp...]
   

After migration, the domain resides on the specified disks and the LSM volume no longer exists.

10.7.5    Unencapsulating Swap Volumes

You can unencapsulate swap devices for a cluster member to move them from LSM volumes onto physical disk partitions.

To unencapsulate all members' swap devices, perform the following steps on all members with encapsulated swap devices:

1. Display the names of LSM volumes to determine the node name for the member and the disk name in the member's swap volume:

   # volprint -vht
   

   In the output, look for:

   • The nodename-swapnn volume name; for example, larry-swap01.

   • The disk name for the swap volume in the form dsknp; for example, dsk10b.

2. Display a list of LSM disks:

   # voldisk list
   

   In the output, look for the private region of the swap disk, in the form dsknp. The disk number (dskn) will be the same as the disk name for the swap volume; for example, dsk10f.

3. Edit the /etc/sysconfigtab file for the member and remove the /dev/vol/rootdg/nodename-swapnn entry from the swapdevice= line.

4. Remove the swap disk's private region partition from the disk group and from LSM control:

   # voldg rmdisk dsknp
   # voldisk rm dsknp
   

5. Reboot the member:

   # shutdown -r now
   

   When the member starts again, it no longer uses the LSM swap volume.

6. Log back in to the same member.

7. Remove the swap volume:

   # voledit -rf rm nodename-swapnn
   

8. Remove the LSM disk for the swap volume from the disk group, and remove the disk from LSM control:

   # voldg rmdisk dsknp
   # voldisk rm dsknp
   

9. Set the cluster member to swap on the disk partition:

   # swapon /dev/disk/dsknp
   

10. Edit the /etc/sysconfigtab file as follows:

    • Add the /dev/disk/dsknp entry to the line swapdevice= so that the line reads:

      swapdevice=/dev/disk/dsknp
      

    • If you removed the last LSM swap device for this member, set the value for lsm_root_dev_is_volume= to 0.

The cluster member uses the specified disk partition for its swap device, and the LSM swap volume no longer exists.