8 Cluster Alias Application Programming Interface

This chapter discusses the following topics:

Cluster alias port terminology (Section 8.1)

Cluster alias functions (Section 8.2)

Cluster port space (Section 8.3)

Information for multi-instance services that bind to reserved ports (Section 8.4)

Cluster alias setsockopt() options (Section 8.5)

Port attributes: the relationship between the port attributes set by /etc/clua_services, clua_registerservice(), and the setsockopt() cluster alias socket options (Section 8.6)

UDP applications and source addresses (Section 8.7)

8.1 Cluster Alias Port Terminology

The following port-related terms are used in this chapter:

well-known port: A port whose number is known by both the client and server. For example, those ports whose numbers are listed in /etc/services. The server explicitly binds to and listens on its well-known port because clients know the port number and expect to connect to the server via its well-known port.
dynamic port: The inverse of a well-known port. The application does not explicitly specify a port number; it is assigned the first open port that is found by the operating system.
ephemeral port: A dynamic port within the ephemeral port space (ports with numbers greater than 1024, or more explicitly, between IPPORT_RESERVED and IPPORT_USERRESERVED).
locked port: A port that is dedicated to an individual cluster member and is not available clusterwide. When a port is locked, an attempt by an application to bind to the port fails with EADDRINUSE, unless the application attempting to bind sets the SO_REUSEALIASPORT option on the socket.
reserved port: In the cluster alias subsystem, a port whose number is greater than 512 and less than IPPORT_RESERVED (1024). The default cluster behavior is that a bind to a reserved port automatically locks the port. (Ports with numbers less than or equal to 512 are never locked.)

8.2 Cluster Alias Functions

The cluster alias library, libclua.a and libclua.so, provides functions for getting and setting cluster alias subsystem attributes. Table 8-1 lists the functions that the cluster alias library provides. See the specific section 3 reference pages for more information.

Table 8-1: Cluster Alias Functions

Function	Description
`clua_error`	Map a cluster alias message ID to a printable character string, returning the string to the caller.
`clua_getaliasaddress`	Get the IP address of one cluster alias known to the local node.
`clua_getaliasinfo`	Get information about one cluster alias and its members.
`clua_getdefaultalias`	Get the IP address of the default cluster alias.
`clua_isalias`	Determine whether an IP address is that of a cluster alias.
`clua_registerservice`	Register a dynamic port as eligible to receive incoming connections.
`clua_unregisterservice`	Release a port.
`clusvc_getcommport`	Bind to a common port within a cluster.
`clusvc_getresvcommport`	Bind to a reserved port within a cluster.
`print_clua_liberror`	Map a cluster alias message ID to a printable character string, returning the string to `stderr`.

Use the following #include files and libraries when writing and compiling programs that use cluster alias functions:

Programs that use the clua functions #include <clua/clua.h> and are compiled with -lclua.

Programs that use the clua_getaliasaddress(), clua_getaliasinfo(), clua_getdefaultalias(), clua_isalias(), clua_registerservice(), or clua_unregisterservice() functions are compiled with -lclua -lcfg.

Programs that use the clusvc functions #include <netinet/in.h>, and are compiled with -lclu.

The following list describes the cluster alias functions in more detail:

clua_error() and print_clua_liberror()

The clua_error() and print_clua_liberror() functions map a cluster alias message ID to a printable character string. The clua_error() function returns the string to the caller. The print_clua_liberror() function prints the string to stderr. See clua_error(3) for more information.

clua_getaliasaddress() and clua_getaliasinfo()

A call to the clua_getaliasaddress function gets the IP address of one cluster alias known to the local node. Each subsequent call returns another alias IP address. When the list of aliases that are known to the node is exhausted, the function returns CLUA_NOMOREALIASES.

When it is passed the address of an alias in a sockaddr structure, clua_getaliasinfo() populates a clu_info structure with information about that alias.

Programs usually make iterative calls to clua_getaliasaddress and pass each alias address to clua_getaliasinfo to get information about that alias. The following example shows this iterative loop (wrapped inside a short main() program with some calls to printf()):

/*  compile with -lclua -lcfg */
#include <sys/socket.h> /* AF_INET */
#include <clua/clua.h>  /* includes <netinet/in.h> */
#include <netdb.h>      /* gethostbyaddr() */
#include <arpa/inet.h>  /* inet_ntoa() */
 
main ()
{
  int context = 0;
  struct sockaddr addr;
  struct clua_info outbuf, *pout;
  clua_status_t result1, result2;
  struct hostent *hp;
  pout=&outbuf;
 
  while ((result1=clua_getaliasaddress(&addr,
                                       &context)) == CLUA_SUCCESS)
   {
     if ((result2=clua_getaliasinfo(&addr, pout)) == CLUA_SUCCESS) {
      hp = gethostbyaddr((const void *)&pout->addr,
                            sizeof (struct in_addr), AF_INET);
        printf ("\nCluster alias name:\t\t %s\n", hp->h_name);
        printf ("Cluster alias IP address:\t %s\n",
inet_ntoa(pout->addr)));
        printf ("Cluster alias ID (aliasid):\t %d\n", pout->aliasid);
        printf ("Connections rcvd from net:\t %d\n",
pout->count_cnx_rcv_net);
        printf ("Connections forwarded:\t\t %d\n",
pout->count_cnx_fwd_clu);
        printf ("Connections rcvd within cluster: %d\n",
                pout->count_cnx_rcv_clu);
     } else {
        print_clua_liberror(result2);
        break;
     }
   }
  if (result1 != CLUA_SUCCESS && result1 != CLUA_NOMOREALIASES)
   print_clua_liberror(result1);
}

See clua_getaliasaddress(3) and clua_getaliasinfo(3) for more information.

clua_getdefaultalias()

Where the clua_getaliasaddress() function can iteratively return the IP addresses of all cluster aliases that are known to the local node, the clua_getdefaultalias() function returns only the address of the default cluster alias. See clua_getdefaultalias(3) for more information.

clua_isalias()

When it is passed an IP address, the clua_isalias() function determines whether the address is that of a cluster alias. See clua_isalias(3) for more information.

clua_registerservice() and clua_unregisterservice()

The clua_registerservice() function registers a dynamic port as eligible to receive incoming connections. For ports in the range 512-1024, use the CLUSRV_STATIC option. Otherwise, the port is reserved clusterwide by the first node to bind to the port and the remaining cluster members will not be able to bind to the port.

The clua_unregisterservice() function releases a port.

See clua_registerservice(3) for more information.

clusvc_getcommport() and clusvc_getresvcommport()

Programs that use RPC can call the clusvc_getcommport() and clusvc_getresvcommport() functions to bind to a common port within a cluster. Use clusvc_getresvcommport() when binding to a reserved (privileged) port, a port number in the range 0-1024. See Section 8.4 for information on binding to reserved ports. The following example shows a typical calling sequence (wrapped inside a short main() program with some calls to printf()):

/* compile with -lclu */
#include <rpc/rpc.h>   /* includes <netinet/in.h> */
#include <syslog.h>    /* LOG_ERR */
#include <unistd.h>    /* gethostname() */
#include <sys/param.h> /* MAXHOSTLEN */
 
main () {
  int s, i, namelen;
  int cluster = 0;
  uint prog = 100999;  /* replace with real program number */
  struct sockaddr_in addr;
  int len = sizeof(struct sockaddr_in);
  char local_host[MAXHOSTNAMELEN +1];
 
  gethostname (local_host, sizeof (local_host) - 1);
  cluster = clu_is_member();
  printf ("\nSystem %s %s a cluster member\n",
          local_host, cluster?"is":"is not");
 
  if ((s = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)) < 0) {
    syslog(LOG_ERR, "socket: %m");
    exit(1);
  }
 
  bzero(&addr, sizeof(addr));
  addr.sin_family = AF_INET;
  addr.sin_addr.s_addr = INADDR_ANY;
 
  if (cluster) {
    if (clusvc_getcommport(s, prog, IPPROTO_UDP, &addr) < 0) {
      syslog(LOG_ERR, "clusvc_getcommport: %m");
      exit(1);
    }
 
  } else {
    addr.sin_port = 0;
    if (bind(s, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
      syslog(LOG_ERR, "bind: %m");
      exit(1);
    }
    if (getsockname(s, (struct sockaddr *)&addr, &len) != 0) {
      syslog(LOG_ERR, "getsockname: %m");
      (void) close(s);
      exit(1);
    }
  }
  printf ("  addr.sin_family: %d\n", addr.sin_family);
  printf ("  addr.sin_port:   %u\n", addr.sin_port);
  printf ("  addr.sin_addr:   %x\n", addr.sin_addr);
  printf ("  addr.sin_zero:   ");
  for (i = 0; i<8;i++)
    printf("%d ", (int)addr.sin_zero[i]);
  putchar('\n');
}

8.3 Cluster Port Space

The cluster emulates single-system port semantics by using the same port space across the cluster. The following list provides a brief overview of how the cluster alias subsystem handles port space.

Ports less than 512 (0 < (IPPORT_RESERVED ^{[Footnote 1]} / 2): These well-known ports are never locked. In a single system, more than one process cannot bind to the same port without using the SO_REUSEPORT socket option. In a cluster, a port whose number is less than 512 is never locked in order to allow one process on each cluster member to bind to the port.

Ports from 512 through 1024 ((IPPORT_RESERVED / 2) to IPPORT_RESERVED): If there is an explicit bind to the port, a reserved port is locked unless the port is marked as static. See Section 8.4 for information on binding to reserved ports.

Ports greater than 1024 (IPPORT_RESERVED to IPPORT_USERRESERVED): An ephemeral port is locked if sin_port=0. An ephemeral port is not locked if there is an explicit bind to the port.

The port space has the following implications for single-instance and multi-instance applications. In general:

Make sure that single-instance applications use a locked port because only one instance of the application should be handling network requests. Therefore, you can use one of the following approaches:
- Explicitly bind to a reserved port (512 through 1024).
- Use an ephemeral port (sin.port=0). (This locks the port, but has the problem that the client must somehow be notified of the port number.)
- Use CAA to make sure that only one instance of the application is running in the cluster.

A multi-instance application does not want to restrict access to the first instance of the application to bind to a port. Therefore, you can use one of the following approaches:
- Use a port that is less than 512.
- Use a reserved port (512 through 1024) and set SO_REUSEALIASPORT.
- Do an explicit bind to an ephemeral port.
- Use an ephemeral port (sin_port=0) and set SO_REUSEALIASPORT. (This keeps the port from being locked, but still has the problem that the client must somehow be notified of the port number.)

8.4 Binding to Reserved Ports (512 Through 1024)

The following information is useful background for multi-instance services binding to reserved ports.

Reserved ports receive special treatment because they can be used as either well-known ports or dynamic ports. By default, if a process explicitly binds to a port in the range 512-1024, the port is assumed to be dynamic: it is reserved clusterwide (locked), and binds on other members will fail. This is primarily done because many programs call bind() looking for free ports in this range, expecting to fail when a port is in use.

The cluster alias subsystem treats reserved ports differently for the following reasons:

There is no way to differentiate between an application that is searching for a dynamic reserved port and one that knows which well-known port it wants.

Dynamic ports are sometimes used as the local port for an outbound connection, which are identified only by port/address pairs. If a dynamic port is used for an outbound connection and if the address is that of a cluster alias, the alias subsystem cannot allow multiple nodes to have the same port. If that were allowed, the alias subsystem would not be able to distinguish between, for example, connection 1000/alias from different nodes in the cluster. Therefore, if the alias subsystem does not know that a reserved port will be static (available clusterwide), it must lock it down.

Ports above 1024 do not encounter this because there is a standard way to allocate a dynamic port: bind to port 0 (sin_port=0) and the system picks a port from the ephemeral port space. Therefore, the cluster alias subsystem assumes that a process that is explicitly binding to a port below 512 or above 1024 knows what it is doing and is getting a well-known port.

Make sure that multi-instance services that use well-known ports in the 512-1024 range for inbound connections register that port as static with the cluster alias subsystem. To register a port as static, either put an entry for the port in /etc/clua_services or modify the program to call clua_registerservice with the CLUASRV_STATIC option. See clua_registerservice(3) and clua_services(4) for more information about these functions. (The static option also serves another purpose for ports greater than 1024: it directs the system to leave a port designated as static out of the ephemeral port space.)

Note

Only specify the static attribute in /etc/clua_services for multi-instance services that are started every time a cluster member boots. For example, services with entries in /etc/inetd.conf. Otherwise, different applications on different cluster members that are looking for a dynamic port might bind to the same port.

8.5 setsockopt() Options

The setsockopt() and getsockopt() system calls support the following cluster alias socket options:

SO_CLUA_DEFAULT_SRC

If the local address has not already been set through a call to bind(), the socket uses the default cluster alias as its source address.

SO_CLUA_IN_NOALIAS

Attempts to bind the socket to a cluster alias address will fail. Use this option for services that you do not want to be accessed using a cluster alias.

A bind to an dynamic port (greater than or equal to IPPORT_RESERVED and less than IPPORT_USERRESERVED) does not result in that port being locked.

A bind to a reserved port with a wildcard address (INADDR_ANY or in6addr_any) does not result in that port being locked.

The source address for outgoing UDP sends or TCP connection requests is a local host address (never a cluster alias address).

The SO_CLUA_IN_NOLOCAL and SO_CLUA_IN_NOALIAS options are mutually exclusive.

SO_CLUA_IN_NOLOCAL

The socket must receive packets that are addressed to a cluster alias and will drop any packets that are not addressed to a cluster alias. Use this option for services that you want only to be accessed using a cluster alias.

The SO_CLUA_IN_NOLOCAL and SO_CLUA_IN_NOALIAS options are mutually exclusive.

SO_RESVPORT

An attempt to bind the socket to a port in the reserved range (512-1024) will fail if the port is marked static, either by a static entry for the port in /etc/clua_services or through a call to clua_registerservice() with the CLUASRV_STATIC option. The call to bind() returns EADDRINUSE.

SO_REUSEALIASPORT

The socket can reuse a locked cluster alias port. When this option is set, a bind() is distributed clusterwide. (A distributed application can use this side effect to determine whether or not a port is in use.)

8.6 Port Attributes: /etc/clua_services, clua_registerservice(), and setsockopt()

The strings that are used in the /etc/clua_services file have a one-to-one mapping with the CLUASRV_* options that are used by clua_registerservice(). Some of these strings/options also have a relationship with the cluster alias setsockopt() options. Table 8-2 lists these relationships.

Table 8-2: Relationship Among Cluster Alias Port Attributes

clua_services	clua_registerservice()	setsockopt()
`in_multi`	`CLUASRV_MULTI`
`in_single`	`CLUASRV_SINGLE`
`in_noalias`	`CLUASRV_IN_NOALIAS`	`SO_CLUA_IN_NOALIAS`
`out_alias`	`CLUASRV_OUT` ^{[Footnote 2]}	`SO_CLUA_DEFAULT_SRC` ^{[Footnote 3]}
`in_nolocal`	`CLUASRV_IN_NOLOCAL`	`SO_CLUA_IN_NOLOCAL`
`static`	`CLUASRV_STATIC`
		`SO_REUSEALIASPORT`
		`SO_RESVPORT`

8.7 UDP Applications and Source Addresses

Some User Datagram Protocol (UDP)-based applications require that messages from the server to the client come from the same server address that is used by the client when sending the UDP request. Because systems in a cluster do not normally use a cluster alias as the source address for outbound UDP messages, these applications may have problems using cluster aliases.

This section describes how to make a UDP-based server application in a cluster respond with the address that was used to reach it (whether the address is a cluster alias or a local address) as follows:

The preferred approach is to bind a socket to each local IP address, plus an additional socket to the default cluster alias address. (Use clua_getdefaultalias to get the alias's address.) Respond to requests using the same socket that received the request; the address will automatically be the one used by the client. This approach replaces using one listen on the wildcard.

A somewhat simpler approach is to use two sockets: one listening on the wildcard and one on the default cluster alias. (As in the previous example, reuse the incoming socket for outbound packets.) If there are multiple local addresses in the same subnet, this approach does not always supply the correct local address, but handles the default cluster alias address properly. If there are multiple cluster aliases and you want the application to respond with the right one (the one used for input), use the multiple sockets approach, with a separate listen() on each cluster alias. Use clua_getaliasaddress() to create a list of defined cluster aliases.

Note

If you want the application to be accessed only using a cluster alias and you want the application to always respond using the default cluster alias, listen on the wildcard with a single socket but set the socket with the SO_CLUA_DEFAULT_SRC option. The application will always use the default cluster alias address for outbound traffic.