2    Hardware Configuration

This chapter provides basic information on how to configure local area network (LAN) hardware for use as a cluster interconnect. It discusses the following topics:

• Configuration guidelines (Section 2.1)

• Supported configurations and configuration examples (Section 2.2)

This chapter focuses on configuring LAN hardware as a cluster interconnect. For full cluster and storage configuration information, see the Cluster Hardware Configuration manual.

2.1    Configuration Guidelines

Any Ethernet adapter, switch, or hub that works in a standard LAN at 100 Mb/s should work within a LAN interconnect.

Note

Fiber Distributed Data Interface (FDDI), ATM LAN Emulation (LANE), 10 Mb/s Ethernet, and Gigabit Ethernet are not supported in a LAN interconnect.

The following features are required of Ethernet hardware participating in a cluster LAN interconnect:

• The LAN interconnect must be private to cluster members. A packet that is transmitted by one cluster member's LAN interconnect adapter can be received only by other members' LAN interconnect adapters.

• A LAN interconnect can be a single direct full-duplex connection between two cluster members or can employ either switches or hubs (but not both). One or more switches are required for a cluster of three or more members and for a cluster whose members use a redundant array of independent network adapters (NetRAIN) virtual interface for their cluster interconnect device.

  Note

  Although hubs and switches are interchangeable in most LAN interconnect configurations, switches are recommended for performance and scalability. Because most hubs run in half-duplex mode, their use in a LAN interconnect may limit cluster performance. Additionally, hubs do not provide the features (described in Appendix A) required for a dual redundant LAN interconnect configuration. Overall, using a switch, rather than a hub, in a LAN interconnect provides greater scalability for clusters with three or more members.

• Adapters and switch ports must be configured compatibly with respect for 100 Mb/s full-duplex operation.. If you are using a switch with any of the DE60x family of adapters (which have a console name of the form eix0), use a switch that supports autonegotiation. If you are using a switch with network adapters in the DE50x family (which have a console name of the form ew x0) that do not autonegotiate properly, the switch must be capable of disabling autonegotiation. (See Section 4.7.1 for a discussion of the symptoms of misconfigured LAN hardware.)

• If you use two crossover cables to link two switches in a fully redundant LAN cluster interconnect (Figure 2-3 and Figure 2-4), you must configure the switches to avoid packet-forwarding problems caused by the routing loop created by the second link. Typical switches provide at least one of the following three mechanisms for support of parallel interswitch links. In order of decreasing desirability for cluster configurations, the mechanisms are:

  Link aggregation

  Treats multiple physical links between a pair of switches as a single link and distributes packet traffic among them.

  Link resiliency

  Treats multiple physical links between a pair of switches as an active link and one or more standby links and fails over between them.

  Spanning Tree Protocol

  Employs a distributed routing algorithm that allows switches to cooperate to discover and remove routing loops.

  See Appendix A for a detailed discussion of the switch requirements and configuration options appropriate to each mechanism.

• Although it may be used to eliminate routing loops on switch ports used for parallel links between switches, Spanning Tree Protocol (STP) must be disabled on all Ethernet switch ports connected to cluster members, whether the members are using single adapters or multiple adapters included in NetRAIN devices. If this is not the case, cluster members will be flooded by broadcast messages which, in effect, create denial-of-service symptoms in the cluster. See Section 4.7.5 for additional information.

• All cluster members must have at least one point-to-point connection to all other members. If the Ethernet adapters that are used for the LAN interconnect fail on a given member, that member loses communication with all other members. A cluster interconnect configuration that requires a member to route interconnect traffic from another member to a different subnet is unsupported. That is, you cannot replace a switch with a member system.

• Up to two switches are allowed between two cluster members. You must not introduce unacceptable latencies by using, for example, a satellite uplink or a wide area network (WAN) as the path between two components of a LAN interconnect.

• Link aggregation of Ethernet adapters using Tru64 UNIX features (including the lagconfig command) is not supported for a LAN interconnect. Link aggregation of adapters is supported for the cluster's external networks.

• To simplify management, configure the LAN interconnect network adapters symmetrically on all cluster members. Installing the same type of adapter in each member in the same relative position with respect to other network adapters helps ensure that the adapters have similar names across cluster members. In a fully redundant LAN interconnect configuration using two or more interconnected switches, and NetRAIN virtual interfaces as member interconnect devices, you should uniformly connect the first network adapter listed in each member's NetRAIN set to the first switch and the second network adapter to the second switch. This simplifies the identification of the adapters for monitoring and maintenance. Additionally, it ensures that the active adapters of each member are connected to the same switch when the cluster is initially booted. As discussed in Section 2.2.3, one method for guarding against a network partition of the cluster in certain failure conditions is to ensure that all active adapters in the LAN interconnect are connected to the same switch.

2.2    Supported LAN Interconnect Configurations

TruCluster Server currently supports up to eight members in a cluster, regardless of whether its cluster interconnect is based on LAN or Memory Channel. Chapter 1 of the Cluster Hardware Configuration manual illustrates some cluster configurations using Memory Channel. The following sections supplement that chapter by discussing the supported LAN interconnect configurations:

• A single crossover cable directly connecting the Ethernet adapter of one member to the Ethernet adapter of a second member (two-member cluster only) (Section 2.2.1)

• A single switch connecting two to eight members (Section 2.2.2)

• Two switches (with one or more crossover cables between them), with two or more Ethernet adapters on each member, configured as a NetRAIN virtual interface but each connected to a different switch (Section 2.2.3)

• Clustered AlphaServer DS10L systems (Section 2.2.4)

2.2.1    Two Cluster Members Directly Connected by a Single Crossover Cable

You can configure a LAN interconnect in a two-member cluster by using a single crossover cable to connect the Ethernet adapter of one member to that of the other, as shown in Figure 2-1. (See Section 3.1.2 for an explanation of the IP addresses shown in the figure.)

Figure 2-1:  Two Cluster Members Directly Connected by a Single Crossover Cable

Note

A crossover cable for point-to-point Ethernet connections is required to directly connect the network adapters of two members when no switch or hub is configured between them.

From a member's perspective, because this cluster does not employ redundant LAN interconnect components (each member has a single Ethernet adapter and a single cable connects the two members), a break in the LAN interconnect connection (for example, the servicing of a member's Ethernet adapter or a detached cable) will cause a member to leave the cluster. However, if you configure a voting quorum disk in this cluster, the cluster itself will survive the failure of either member or of the quorum disk, or a break in the LAN interconnect connection. Similarly, if you configure one member with a vote and the other with no votes, the cluster will survive the failure of the nonvoting member or of its LAN interconnect connection.

You can expand this configuration by adding a switch between the two members. A switch is required in the following cases:

• When the cluster expands beyond two members (for example, the configuration discussed in Section 2.2.2).

• When you add a second Ethernet adapter to each member in order to configure the cluster interconnect device as a NetRAIN virtual interface. Merely adding the second adapters and a second crossover cable link does not provide the connectivity required for NetRAIN failover in all circumstances and is not supported.

2.2.2    Cluster Using a Single Ethernet Switch

You can configure a cluster with a single Ethernet hub or switch connecting two through eight members. For optimal performance, we recommend a switch for clusters of three or more members.

Any member that has multiple Ethernet adapters can configure them as a NetRAIN set to be used as its LAN interconnect interface. Doing so allows those members to remain cluster members even if they lose one internal connection to the LAN interconnect.

The three-member cluster in Figure 2-2 uses a LAN interconnect incorporating a single Ethernet switch. Each member's cluster interconnect is a NetRAIN virtual interface consisting of two network adapters. (See Section 3.1.2 for an explanation of the IP addresses shown in the figure.)

Figure 2-2:  Three-Member Cluster Using a Single Ethernet Switch

Assuming that each member has one vote, this cluster can survive the failure of a single member or a single break in a member's LAN interconnect connection (for example, the servicing of an Ethernet adapter or a detached cable). From a member's perspective, any member can survive a single break in its LAN interconnect connection. However, the servicing or failure of the switch will make the cluster nonoperational. The switch remains a single point of failure in a cluster of any size, except when it is used in one of the recommended two-member configurations using a quorum disk discussed in Section 2.2.1. For this reason, the cluster in Figure 2-2 is not a recommended configuration.

By adding a second switch to this cluster, and connecting a LAN interconnect adapter from each member to each switch (as discussed in Section 2.2.3), you can eliminate the switch as a single point of failure and increase cluster reliability.

2.2.3    Cluster Using Fully Redundant LAN Interconnect Hardware

You can achieve a fully redundant LAN interconnect configuration by using NetRAIN and redundant paths from each member through interconnected switches. In the four-member cluster in Figure 2-3 and Figure 2-4, two Ethernet adapters on each member are configured as a NetRAIN virtual interface, two switches are interconnected by two crossover cables, and the Ethernet connections from each member are split across the switches.

Figure 2-3:  Recommended Fully Redundant LAN Interconnect Configuration Using Link Aggregation or Link Resiliency

Figure 2-4:  Recommended Fully Redundant LAN Interconnect Configuration Using the Spanning Tree Protocol

Note

If you are mixing switches from different manufacturers, consult with your switch manufacturers for compatibility between them.

Like the three-member cluster discussed in Section 2.2.2, this cluster can tolerate the failure of a single member or a single break in a member's LAN interconnect connection (for example, the servicing of an Ethernet adapter or a detached cable). (This assumes that each member has one vote and no quorum disk is configured.) However, this cluster can also survive a single switch failure and the loss of the crossover cables between the switches.

Because NetRAIN must probe the inactive LAN interconnect adapters across switches, the crossover cable connection between the switches is important. Two crossover cables are strongly recommended. When two crossover cables are used, as shown in Figure 2-3 and Figure 2-4, the loss of one of the cables is transparent to the cluster. As discussed in Appendix A, when using parallel interswitch links in this manner, it is important to employ one of the methods provided by the switches for detecting or avoiding routing loops between the switches. These figures indicate the appropriate port settings with respect to the most common methods provided by switches: link aggregation (also known as port trunking), link resiliency (both shown in Figure 2-3), and Spanning Tree Protocol (STP) (shown in Figure 2-4). (See Section 3.1.2 for an explanation of the IP addresses shown in the figure.)

In some circumstances (like the nonrecommended configuration, shown in Figure 2-5, that uses a single crossover cable), a broken crossover connection can result in a network partition. If the crossover connection is completely broken, its loss prevents NetRAIN from sending packets to the inactive adapters across the crossover connection. Although this situation will not cause the cluster to fail, it will disable failover between the adapters in the NetRAIN sets.

For example, in the configuration shown in Figure 2-5 the active LAN interconnect adapters of Members 1 and 2 are currently on Switch 1; those of Members 3 and 4 are on Switch 2. If the crossover connection is broken while the cluster is in this state, Members 1 and 2 can see each other but cannot see Members 3 and 4 (and thus will remove them from the cluster). Members 3 and 4 can see each other but cannot see Members 1 and 2 (and thus will remove them from the cluster). By design, neither cluster can achieve quorum; each has two votes out of a required three, and both will hang in quorum loss.

Figure 2-5:  Nonrecommended Redundant LAN Interconnect Configuration

To decrease a cluster's vulnerability to network partitions in a dual-switched configuration, take any or all of the following steps:

• Configure the cluster with two crossover cables between the switches, as shown in Figure 2-3. This configuration reduces vulnerability to a network partition, but requires that the switches be additionally configured to avoid packet-forwarding problems caused by the routing loop created by the second link. See Appendix A for a detailed discussion of the switch requirements and configuration mechanisms.

• To avoid a cluster hang due to quorum loss that can occur when a cluster encounters a network partition, configure the cluster with an odd number of votes, either by providing an odd number of voting members or a voting quorum disk.

• After performing network maintenance (for example, when replacing cables or adapters) or at any other time you believe that NetRAIN failover has occurred, examine the connectivity of the active network adapters on the NetRAIN devices on the cluster interconnect. On each member, issue an ifconfig command on the LAN interconnect's NetRAIN virtual interface to ensure that the active LAN interconnect adapter on each member is connected to the same switch. Uniformly connecting each member's first network adapter to the first switch and its second network adapter to the second switch facilitates identifying the member adapters that are connected to a given switch. If the active adapters are split across the switches, use the ifconfig nrx switch command, as appropriate, to consolidate them on a single switch.

2.2.4    Clustering AlphaServer DS10L Systems

Support for the LAN interconnect makes it possible to cluster more basic AlphaServer systems, such as the Compaq AlphaServer DS10L. The AlphaServer DS10L is an entry-level system that ships with two 10/100 Mb/s Ethernet ports, one 64-bit PCI expansion slot, and a fixed internal IDE disk. The 44.7 x 52.1 x 4.5-centimeter (17.6 x 20.5 x 1.75-inch (1U)) size of the AlphaServer DS10L, and the ability to rackmount large numbers of them in a single M-series cabinet, make clustering them an attractive option, especially for Web-based applications.

When you configure an AlphaServer DS10L in a cluster, we recommend that you use the single PCI expansion slot for the host bus adapter for shared storage (where the cluster root, member boot disks, and optional quorum disk reside), one Ethernet port for the external network, and the other Ethernet port for the LAN interconnect. Figure 2-6 shows a very basic low-end cluster of this type consisting of four AlphaServer DS10Ls.

Figure 2-6:  Low-End AlphaServer DS10L Cluster

Although the configuration shown in Figure 2-6 represents an inexpensive and useful entry-level cluster, its LAN interconnect and shared SCSI storage bus present single points of failure. That is, if the shared storage bus or the LAN interconnect switch fails, the cluster becomes unusable.

To eliminate these single points of failure, the configuration in Figure 2-7 adds two AlphaServer ES40 members to the cluster, plus two parallel interswitch connections. Two AlphaServer DS10L members are connected via Ethernet ports to one switch on the LAN interconnect; two are connected to the other switch. A Fibre Channel fabric employing redundant Fibre Channel switches replaces the shared SCSI storage in the previous configuration.

Although not distinctly shown in the figure, the host bus adapters of two DS10Ls are connected to one Fibre Channel switch; those of the other two DS10Ls are connected to the other Fibre Channel switch.

Figure 2-7:  Cluster Including Both AlphaServer DS10L and AlphaServer ES40 Members

The physical LAN interconnect device on each of the two AlphaServer ES40 members consists of two Ethernet adapters configured as a NetRAIN virtual interface. On each ES40, one adapter is cabled to the first Ethernet switch and the other is cabled to the second Ethernet switch. Similarly, each ES40 contains two host bus adapters connected to the Fibre Channel fabric. On each, one adapter is connected to the first Fibre Channel switch, the other is connected to the second Fibre Channel switch.

When delegating votes in this cluster, you have a number of possibilities:

• Assign one vote to each AlphaServer ES40 member and no votes to the AlphaServer DS10L members. Configure a quorum disk with a vote on the shared storage. This cluster can survive the loss of any one AlphaServer ES40 member, the quorum disk, or any or all AlphaServer DS10L members.

• Assign one vote to each member. Configure a quorum disk with a vote on the shared storage. This cluster can survive the loss of one or both of the AlphaServer ES40 members or the loss of three DS10L members. (In other words, the AlphaServer ES40 members require the votes of at least one AlphaServer DS10L member, plus the quorum disk vote, to maintain quorum.)