11    Configuring an Eight-Member Cluster Using Externally Terminated Shared SCSI Buses

This chapter discusses the following topics:

• Overview of an eight-node cluster (Section 11.1)

• How to configure an eight-node cluster using an UltraSCSI BA356 and external termination (Section 11.2)

TruCluster Server Version 5.1A supports eight-member cluster configurations as follows:

• Fibre Channel: Eight-member systems may be connected to common storage over Fibre Channel in a fabric (switch) configuration.

• Parallel SCSI: Only four of the member systems may be connected to any one SCSI bus, but you can have multiple SCSI buses connected to different sets of nodes, and the sets of nodes may overlap. We recommend you use a DS-DWZZH-05 UltraSCSI hub with fair arbitration enabled when connecting four-member systems to a common SCSI bus using RAID array controllers.

  Note

  The DS-DWZZH-03/05 UltraSCSI hubs cannot be connected to a StorageWorks BA35X storage shelf because the storage shelf does not provide termination power to the hub.

Configuring an eight-member cluster using Fibre Channel is straightforward; connect the member systems to the Memory Channel hub and to the Fibre Channel switches. (See Chapter 6 for more information on configuring Fibre Channel).

Configuring an eight-member cluster using shared SCSI buses is more complex because you can only have four member systems on a single shared SCSI bus.

The primary focus of this chapter is on an eight-node cluster that uses externally terminated shared SCSI buses with minimal storage. This type of cluster is of primary interest to high-performance technical computing (HPTC) cluster customers. It is also of importance to those customers who use Tru64 UNIX Versions 4.0F or 4.0G with the TruCluster Software Products Memory Channel Software Version 1.6 product who want to upgrade to Tru64 UNIX Version 5.1A and TruCluster Server Version 5.1A.

11.1    Overview of an Eight-Node TruCluster Server Cluster

Figure 11-1 shows a basic block diagram of an eight-node cluster.

Note

The public network is not shown in Figure 11-1, or in any other illustration in this chapter. Ensure that you have network adapters for your public network.

This is just one of many ways to configure an eight-node cluster. You must choose a configuration that best fits your applications and needs.

Figure 11-1:  Block Diagram of an Eight-Node Cluster

Figure 11-1 shows the following:

• All member systems are connected via the Memory Channel at the Memory Channel hub.

• There are three shared SCSI buses with shared storage on each shared bus:

  • Member systems 1 and 2 are on the first shared SCSI bus.

    The Tru64 UNIX Version 5.1A operating system is installed on member system 1. It can be installed on an internal disk, as is the case in Figure 11-1, or on a shared disk.

    Member system 1 is used to create the cluster with the clu_create command.

    Member system 2 is added to the cluster with the clu_add_member command.

    The shared storage for member systems 1 and 2 contains the root (/), /usr, /var file systems for the cluster, and the boot disks for member systems 1 and 2. (See the Tru64 UNIX Installation Guide and the TruCluster Server Cluster Installation manual for information about installing the Tru64 UNIX and TruCluster Server software.)

    Give member systems 1 and 2 one vote.

  • Member systems 2, 3, 4, and 5 share storage on the second shared SCSI bus. Four systems is the maximum number of cluster members that may be on a shared SCSI bus.

    The shared storage on this bus contains the member system boot disks for member systems 3, 4, and 5.

    Use member system 1 or 2 to add member systems 3, 4, and 5 to the cluster.

    Give member system 3 one vote.

  • Member Systems 1, 6, 7, and 8 form the third shared SCSI bus.

    The shared storage on this bus contains the member system boot disks for member systems 6, 7, and 8.

    Use member system 1 or 2 to add member systems 6, 7, and 8 to the cluster.

Section 11.2.1 and Figure 11-2 provide the details about cabling member systems 1 and 2 for the first shared SCSI bus.

Section 11.2.2 and Figure 11-3 provide the details about cabling the second shared SCSI bus and member systems 2, 3, 4, and 5 into the cluster.

Section 11.2.3 and Figure 11-4 provide the details about cabling the third shared SCSI bus and member systems 1, 6, 7, and 8 into the cluster.

Note

You can install Tru64 UNIX and TruCluster Server Version 5.1A after you complete member system 1 and 2 hardware installation, or you can wait until the hardware for all the systems is installed.

11.2    Configuring an Eight-Node Cluster Using an UltraSCSI BA356 and External Termination

Configuring an eight-node cluster is carried out in three distinct stages, one stage for each shared SCSI bus:

1. Install member systems 1 and 2 and all associated cluster hardware needed to place these two systems on a shared SCSI bus.

2. Install member systems 3, 4, and 5 and all associated cluster hardware needed to place these two systems on a shared SCSI bus with member system 2.

3. Install member systems 6, 7, and 8 and all associated cluster hardware needed to place these two systems on a shared SCSI bus with member system 1.

Note

You can switch steps 2 and 3 around and install member systems 6, 7, and 8 before member systems 3, 4, and 5.

11.2.1    Cabling the First Two Nodes on the First Externally Terminated Shared SCSI Cluster

This section provides installation instructions for the cluster hardware for the first two nodes of an eight-node shared SCSI bus cluster. Complete the steps in order. When you are referred to another section or table, complete those steps completely before returning to this section.

Notes

If you are upgrading from Tru64 UNIX Version 4.0F or V4.0G and TruCluster Memory Channel Software Version 1.6 to Tru64 UNIX Version 5.1A and TruCluster Server Version 5.1A, you are required to have shared storage, but you do not have to change your Memory Channel or public network hardware. Therefore, you can skip those steps except as follows.

If you are using the Memory Channel adapters in multiple-active rail mode with the TruCluster Memory Channel Software product, after you have installed the Tru64 UNIX and TruCluster Server Version 5.1A software, you will have to reset the rm kernel subsystem configuration rm_rail_style variable to zero. The default for rm_rail_style for TruCluster Server Version 5.1A is 1, which enables failover pair. See the Cluster Highly Available Applications manual for more information.

Figure 11-2 provides a detailed illustration of the first two systems in an 8-node shared SCSI cluster. Table 11-1 lists the components that are used to create the portion of the cluster that is shown in Figure 11-2.

To install the cluster hardware for the first two member systems of an eight-node cluster, follow these steps:

1. Install Memory Channel adapters on member systems 1 and 2. See Chapter 5 for installation and jumper information on the Memory Channel adapters. Delay testing the Memory Channel until you have installed all hardware.

2. Install a Memory Channel hub within 10 meters (32.8 feet) of all eight member systems.

3. Use BN39B-04 (4 meters; 13.1 feet) or BN39B-10 (10 meters; 32.8 feet) to connect the Memory Channel adapters of member systems 1 and 2 to the Memory Channel hub.

4. Refer to the hardware manuals and install the network adapters for the public network on member systems 1 and 2. The public network is not shown in the illustrations in this chapter.

5. Refer to Table 9-3 and install two KZPBA-CB host bus adapters on member system 1 and 2 for the shared SCSI buses that they will use:

   • A shared SCSI bus for member system 1 and 2.

   • A shared SCSI bus for member system 2 with member systems 3, 4, and 5.

   • A shared SCSI bus for member system 1 with member systems 5, 6, and 7.

   Ensure that you set the host bus adapter SCSI IDs as follows:

   • Member system 1: SCSI bus ID 7 (for both host bus adapters)

   • Member system 2: SCSI bus ID 6 (for both host bus adapters)

6. Ensure that each system (member system 1 and 2) has a BN21W-0B Y cable attached to each KZPBA-CB host bus adapter and an H879-AA HD68 terminator attached to one leg of each BN21W-0B Y cable. Member systems 1 and 2 will be at one end of each of the two SCSI buses they share.

7. Prepare the UltraSCSI BA356 for TruCluster Server use (see Section 10.4.1.3). Ensure that you have installed an H8861-AA VHDCI trilink connector on the UltraSCSI BA356 personality module.

   Note

   If you need more storage than one UltraSCSI BA356 provides, you can daisy-chain two of them together. See Section 10.4.3.3 for more information.

8. Select one KZPBA-CB host bus adapter on each system. Connect a BN38C, BN38D, or a combination of a BN38E-0B technology adapter cable and a BN37A cable between the open leg of the BN21W-0B Y cable on each system to the H8861-AA VHDCI trilink connector on the UltraSCSI BA356 personality module. This creates the shared SCSI bus between member systems 1 and 2.

   The remaining KZPBA-CB on each system has an open leg on its BN21W-0B Y cable. These connections will be used for the other shared SCSI buses.

Figure 11-2:  First Two Nodes of an Eight-Node Cluster

Table 11-1:  Hardware Components Used for Configuration Shown in Figure 11-2

If you have performed each step correctly, each of the first two member systems is prepared to be added to three other member systems on a shared SCSI bus.

You can install Tru64 UNIX and TruCluster Server Version 5.1A software at this time, or you can wait until all cluster hardware is installed.

You need to configure two four-node shared SCSI buses to create your eight-node shared SCSI cluster. The next two sections cover the steps needed to configure member systems 3, 4, and 5 on a shared SCSI bus with member system 2, and member systems 6, 7, and 8 on a shared SCSI bus with member system 1.

11.2.2    Cabling the Second Externally Terminated Shared SCSI Bus

So far, you have configured a two-node externally terminated shared SCSI bus made up of member systems 1 and 2. This section covers the steps needed to configure member systems 3, 4, and 5 on a shared SCSI bus with member system 2.

Figure 11-3 shows a detailed illustration of member systems 2, 3, 4, and 5 on the second shared SCSI bus. Table 11-2 shows the components needed to configure the systems shown in Figure 11-3 into the cluster.

To configure member systems 2, 3, 4, and 5 on the second four-node shared SCSI bus, follow these steps:

1. Install the Memory Channel adapters on member systems 3, 4, and 5. See Chapter 5 for installation and jumper information on the Memory Channel adapters. Delay testing the Memory Channel until you have installed all hardware.

   Note

   If member systems 1 and 2 are running cluster software, do not run mc_cable Memory Channel diagnostics. Shut all systems down to the console level to run the mc_cable diagnostic.

2. Use BN39B-04 (4 meters; 13.1 feet) or BN39B-10 (10 meters; 32.8 feet) to connect the Memory Channel adapters of member systems 3, 4, and 5 to the Memory Channel hub.

3. Refer to the hardware manuals and install the network adapters for the public network on member systems 3, 4, and 5. The public network is not shown in the illustrations in this chapter.

4. Referring to Table 9-3, install a KZPBA-CB host bus adapter on member systems 3, 4, and 5. These host bus adapters will be used to form a shared SCSI bus with member system 2.

   Ensure that you set the host bus adapter SCSI IDs as follows:

   • Member system 2: SCSI ID 6 (which should already be set)

   • Member system 3: SCSI ID 7

   • Member system 4: SCSI ID 5

   • Member system 5: SCSI ID 4

5. Ensure that each system (member system 3, 4, and 5) has a BN21W-0B Y cable attached to the KZPBA-CB host bus adapter.

6. Ensure that there is an H879-AA terminator attached to one leg of the BN21W-0B on member system 5. Member systems 2 and 5 will be at the end of this shared SCSI bus.

7. Prepare the UltraSCSI BA356 for TruCluster Server use (see Section 10.4.1.3). Ensure that you have installed an H8861-AA VHDCI trilink connector on the UltraSCSI BA356 personality module.

   Note

   If you need more storage than one UltraSCSI BA356 provides, you can daisy-chain two of them together. See Section 10.4.3.3 for more information.

8. Connect a BN21K, BN21L, or 328215-00X cable between the BN21W-0B Y cables on member system 2 and member system 3.

9. Connect a BN21K, BN21L, or 328215-00X cable between the BN21W-0B Y cables on member system 4 and member system 5.

10. Connect a BN38C, BN38D, or a combination of a BN38E-0B technology adapter cable and a BN37A cable between the open leg of the BN21W-0B on member systems 3 and 4 to the H8861-AA VHDCI trilink connector on the UltraSCSI BA356 personality module.

Figure 11-3:  Second Shared SCSI Bus of an Eight-Node Cluster

Table 11-2 lists the components that are used to create the cluster shown in Figure 11-3.

Table 11-2:  Hardware Components Used for Configuration Shown Figure 11-3

11.2.3    Cabling the Third Externally Terminated Shared SCSI Bus

So far, you have configured a two-node externally terminated shared SCSI bus made up of member systems 1 and 2, and an externally terminated four-node shared SCSI bus with member systems 2, 3, 4, and 5. You need to configure a third externally terminated four-node shared SCSI bus to complete your eight-node shared SCSI cluster.

This section covers the steps needed to configure member systems 1, 6, 7, and 8 on an externally terminated shared SCSI bus.

Figure 11-4 shows a detailed illustration of member systems 1, 6, 7, and 8 on a shared SCSI bus. Table 11-3 lists the components needed to configure the systems shown in Figure 11-4.

To configure member systems 1, 6, 7, and 8 on a four-node shared SCSI bus, follow these steps:

1. Install the Memory Channel adapters on member systems 6, 7, and 8. See Chapter 5 for installation and jumper information on the Memory Channel adapters. Delay testing the Memory Channel until you have installed all hardware.

   Note

   If member systems 1 and 2 are running cluster software, you should not run mc_cable Memory Channel diagnostics. Shut all systems down to the console level to run the mc_cable diagnostic.

2. Use BN39B-04 (4 meters; 13.1 feet) or BN39B-10 (10 meters; 32.8 feet) to connect the Memory Channel adapters of member systems 6, 7, and 8 to the Memory Channel hub.

3. Refer to the hardware manuals and install the network adapters for the public network on member systems 6, 7, and 8. The public network is not shown in the illustrations in this chapter.

4. Referring to Table 9-3, install a KZPBA-CB host bus adapter on member system 6, 7, and 8. These host bus adapters will be used to form a shared SCSI bus with member system 1.

   Ensure that you set the host bus adapter SCSI IDs as follows:

   • Member system 1: SCSI bus ID 7 (which should already be set)

   • Member system 6: SCSI bus ID 6

   • Member system 7: SCSI bus ID 5

   • Member system 8: SCSI bus ID 4

5. Ensure that each system (member system 6, 7, and 8) has a BN21W-0B Y cable attached to the KZPBA-CB host bus adapter.

6. Ensure that there is an H879-AA terminator attached to one leg of the BN21W-0B on member system 8. Member systems 1 and 8 will be at the end of this shared SCSI bus.

7. Prepare the UltraSCSI BA356 for TruCluster Server use (See Section 10.4.1.3.) Ensure that you have installed an H8861-AA VHDCI trilink connector on the UltraSCSI BA356 personality module.

   Note

   If you need more storage than one UltraSCSI BA356 provides, you can daisy-chain two of them together. See Section 10.4.3.3 for more information.

8. Connect a BN21K, BN21L, or 328215-00X cable between the BN21W-0B Y cables on member system 1 and member system 6.

9. Connect a BN21K, BN21L, or 328215-00X cable between the BN21W-0B Y cables on member system 7 and member system 8.

10. Connect a BN38C, BN38D, or a combination of a BN38E-0B technology adapter cable and a BN37A cable between the open leg of the BN21W-0B on member systems 6 and 7 to the H8861-AA VHDCI trilink connector on the UltraSCSI BA356 personality module.

Figure 11-4:  Third Shared SCSI Bus of an Eight-Node Cluster

Table 11-3 lists the components that are used to create the cluster shown in Figure 11-4.

Table 11-3:  Hardware Components Used for Configuration Shown in Figure 11-4