![]() |
|||
![]() |
![]() ![]() |
![]() |
![]() ![]() |
![]() |
![]() ![]() |
![]() |
| ||
Chapter 8Cluster Membership ManagerFor information about how the cluster membership is managed and configured, and how the presence of peer nodes is monitored, see the following sections: Introduction to the Cluster Membership ManagerThe Cluster Membership Manager (CMM) is implemented by the nhcmmd daemon. There is a nhcmmd daemon on each peer node. The nhcmmd daemon on the master node has the current view of the cluster configuration. It communicates its view to the nhcmmd daemons on the other peer nodes. The nhcmmd daemon on the master node determines which nodes are members of the cluster, and assigns roles and attributes to the nodes. It detects the failure of nodes and configures routes for reliable transport. The nhcmmd daemon on the vice-master node monitors the status of the master node. If the master node fails, the vice-master node is able to take over as the master node. The nhcmmd daemons on the master-ineligible nodes do not communicate with one another. Each nhcmmd daemon exports an API to do the following:
Notification messages describe the change and the nodeid of the affected node. Clients can use notifications to maintain an accurate view of the peer nodes in the cluster. For further information about the nhcmmd daemon, see the nhcmmd(1M) man page. You can use the CMM API to write applications that manage peer nodes or that register clients to receive notifications. For further information about writing applications that use the CMM API, see the Netra High Availability Suite Foundation Services 2.1 6/03 CMM Programming Guide. Configuring the Cluster MembershipCluster membership information is stored in the configuration files, cluster_nodes_table and nhfs.conf. At cluster startup, the cluster membership is configured as follows:
The nhcmmd daemon on the master node updates its cluster_nodes_table and its view of the cluster configuration when a peer node is added, removed, or disqualified. The nhcmmd daemon on the master node communicates the updated view to the nhcmmd daemons on the other peer nodes. The vice-master node uses this view to update its local cluster_nodes_table. In this way, the master node and vice-master node always have an up-to-date view of the cluster. Monitoring the Presence of Peer NodesEach peer node runs a daemon called nhprobed that periodically sends a heartbeat in the form of an IP packet. Heartbeats are sent through each of the two physical interfaces of each peer node. When a heartbeat is detected through a physical interface, it indicates that the node is reachable and that the physical interface is alive. If a heartbeat is not detected for a period of time exceeding the detection delay, the physical interface is considered to have failed. If both of the node's physical interfaces fail, the node itself is considered to have failed. The detection delay is 900 milliseconds. At least one heartbeat must be detected each 900 milliseconds. For more information about the nhprobed daemon, see the nhprobed(1M) man page. Interaction Between the nhprobed Daemon and the nhcmmd DaemonOn the master-eligible nodes, the nhprobed daemon receives a list of nodes from the nhcmmd daemon. The nhprobed daemon monitors the heartbeats of the nodes on the list. On the master node, the list contains all of the master-ineligible nodes and the vice-master node. On the vice-master node, the list contains the master node only. On the master-eligible nodes, the nhprobed daemon notifies the nhcmmd daemon when, for any node on its list, any of the following events occur:
When a node other than the master node becomes unavailable, the master node eliminates the node from the cluster. The master node uses the TCP abort facility to close communication to the node. When the master node becomes unavailable, a failover is provoked. Using the Direct Link to Prevent Split Brain ErrorsSplit brain is an error scenario in which the cluster has two master nodes. A direct communication link between the master-eligible nodes prevents the occurrence of split brain when the communication between the master node and vice-master node fails. As described in Monitoring the Presence of Peer Nodes, the nhprobed daemon on the vice-master node monitors the presence of the master node. If the nhprobed daemon on the vice-master node fails to detect the master node, the master node itself or the communication to the master node has failed. If this happens, the vice-master node uses the direct link to try to contact the master node.
The Node Management Agent can monitor the following statistics on the direct link:
For information about how to connect the direct link between the master-eligible nodes, see the Netra High Availability Suite Foundation Services 2.1 6/03 Hardware Guide. Multicast Transmission of HeartbeatsProbe heartbeats are multicast. Each cluster on a local area network (LAN) is assigned to a different multicast group, and each network interface card (NIC) on a node is assigned to a different multicast group. For example, NICs connected to an hme0 Ethernet network are assigned to one multicast group, and NICs connected to an hme1 Ethernet network are assigned to another multicast group. A heartbeat sent from one multicast group cannot be detected by another multicast group. Therefore, heartbeats sent from one cluster cannot be detected by another cluster on the same LAN. Similarly, for a cross-switched topology, heartbeats sent from one Ethernet network cannot be detected on another Ethernet network. Multicast addresses are 32-bit. The lower 28 bits of the multicast address represent the multicast group. The multicast address is broken into the following parts:
| ||
| ||
![]() |