Frontend nodes automatically connect to another router if the one being used fails. This reconnection is transparent to the application.

Routers are responsible for coordinating the two-phase commit for transactions. If the original router coordinating a transaction fails, backend nodes select another router that can ensure correct transaction completion.

1.6.5.1 Backend Restart Recovery

Transactions in the process of being committed at the time of a failure are recovered from RTR's disk journal. Recovery could be with a concurrent server, a standby server, or a restarted server created when the failed backend restarts.

Correct ordering of the execution of transactions against the database is maintained.

1.6.5.2 Transaction Message Replay

Transaction messages which are lost in transit are re-sent when possible. The frontend and backend nodes keep an in-memory copy of all active messages for this purpose.

1.6.5.3 Link Failure Recovery

In the event of a communications failure, RTR tries to reconnect the link or links until it succeeds.

1.7 Flexibility and Growth

RTR allows you to cope easily with changes in:

• Network demand
• User access patterns
• The volume of data

Since an RTR-based system can be built using multiple systems at each functional layer, it easily lends itself to step-by-step growth, avoiding unused capacity at each stage. With your system still up and running, it is possible to:

• Create and delete concurrent server processes.
• Add or remove nodes (frontend, router or backend).

This means you do not need to provide spare capacity to allow for growth.

RTR also allows parallel execution. This means that different parts of a single transaction can be processed in parallel by multiple servers.

RTR provides a comprehensive set of monitoring tools to help you evaluate the volume of traffic passing through the system. This can help you can respond to unexpected load changes by altering the system configuration dynamically.

1.8 Failure Scenarios

This section describes how RTR recovers from different hardware and software failure. For more information on failure and recovery scenarios, see the RTR Application Design Guide.

1.8.1 Loss of a Backend CPU

If standby or shadow servers are available on another backend node, operation of the rest of the system will continue without interruption, using the standby or shadow server.

If a backend processor is lost, any transactions in progress are recovered either when the backend restarts, or by a standby if one is present. This means that the distributed database will be brought back to a transaction-consistent state.

1.8.2 Loss of a Router

If a router fails and another router node is available, all in-progress transactions are transparently re-routed by the other router. System operation will continue without interruption.

1.8.3 Loss of a Frontend

If a frontend is lost:

• All transactions committed but not completed on the frontend node at the time of failure will be completed.
• All transactions started but not committed on the frontend node at the time of failure will be aborted.

The RTR application programming interface (API) that is provided with Reliable Transaction Router is identical on all hardware and operating system platforms that support RTR. This API (sometimes referred to as the Portable API) is described in the following chapter.

In addition, a command line interface (CLI) to the API is available. This enables you to write simple RTR applications for testing. The RTR CLI is described in the RTR System Manager's Manual.

The term Portable API is used to describe the RTR application programming interface (API) adopted in Reliable Transaction Router Version 3; this API is available on all platforms on which Reliable Transaction Router is supported.

2.1 Transactional Messages

RTR allows the client and server applications to communicate by entering into a dialogue consisting of an exchange of messages between a client application (the dialogue initiator) and one or more server applications.

Each dialogue forms a transaction in which all participants have the opportunity to either accept or reject the whole transaction. When the transaction is complete, all participants are informed of the transaction's completion status: success (rtr_mt_accepted) if all participants accepted it, failure (rtr_mt_rejected) if any participant rejected it. (For more information on messages, see Section 2.10, RTR Messages.)

2.2 RTR Channels

With RTR, applications can be engaged in several transactions at a time.

To support many in-progress transactions at the same time, RTR lets applications open multiple channels. An application opens one or more channels to RTR, and any transaction is associated with only one channel. The transaction is said to be active on that channel. For example, a client application opens a channel and then sends the first message of a transaction on that channel. All messages sent and received for that transaction are now associated with that channel.

While waiting for a response from the server, the client application can open a second channel and start a new transaction on it. When the transaction on the first channel has completed, the client application may start the next transaction on it, or simply issue the rtr_accept_tx call.

Similarly, a server application may open several channels and, when the first message of a new transaction arrives, RTR delivers it on the first available channel. That channel remains associated with the transaction until it completes.

An application opens a channel before it can send or receive messages; the RTR API call rtr_open_channel is used to do this. The RTR call specifies whether the channel is a client channel or a server channel; it cannot be both. (This restriction helps to simplify application structure, and each channel type has some special properties.) A single application can, however, open client channels and server channels.

2.3 Broadcast Messages and Events

In addition to transactional messages, client or server programs may broadcast event messages. These are delivered to some subset of the distributed applications, as specified by the event-number and event-name parameters. In contrast to transactional dialogues, no completion status is subsequently returned to the initiator. A message can be from 0 to 64K bytes long.

Both client and server channels receive messages from RTR. Additionally a client channel receives event messages from servers, and a server channel receives event messages from clients. To enable a client to receive event messages from a client, the client can open a server channel, and similarly for a server. Events are more fully described in Section 2.11, RTR Events.

2.4 Portable API Calls

The Portable API calls are shown in Table 2-1, Portable API Calls. Each call is shown with a brief description and whether it can be used on client channels or server channels or both. Calls are listed in alphabetical order.

Table 2-1 Portable API Calls

RTR Call	Description	Channel Use
rtr_accept_tx	Accepts a transaction	Client and server
rtr_broadcast_event	Broadcasts (sends) an event message	Client and server
rtr_close_channel	Closes a previously opened channel	Client and server
rtr_error_text	Gets the text for an RTR status number	Client and server
rtr_get_tid	Gets the current transaction ID	Client and server
rtr_open_channel	Opens a channel for sending and receiving messages	Client and server
rtr_prepare_tx	Prepares a nested transaction to be committed	Client only
rtr_receive_message	Receives the next message (transaction message, event or completion status)	Client and server
rtr_reject_tx	Rejects a transaction	Client and server
rtr_reply_to_client	Sends a response from a server to a client	Server only
rtr_request_info	Requests information from RTR	Client and server
rtr_send_to_server	Sends a message from a client to the server(s)	Client only
rtr_set_info	Sets an RTR parameter	Client and server
rtr_set_user_handle	Associates a user value with a transaction	Client and server
rtr_set_wakeup	Sets a function to be called on message arrival	Client and server
rtr_start_tx	Explicitly starts a transaction	Client only

2.5 Programming Examples

The following pseudocode examples of a client and a server application illustrate the use of the Portable API. Details have been omitted to keep the basic structure clear.

2.5.1 Simple Client

This simple client program issues transactions and receives event messages. It simply issues one transaction, waits for it to be processed, and in the meantime handles any events that arrive. It then issues the next transaction. It does not need to wait until one transaction finishes before starting the next.

The following two examples are single-threaded. They can be made multithreaded by opening more channels. The structure of the main receive loop does not need to be changed to implement this. Note that rtr_receive_message receives the next message in the process input queue for any of the channels opened by the program (unless preferred channels have been requested in the rtr_receive_message .

Example 2-1 Example Client

rtr_open_channel() ! Open a channel to the required facility rtr_receive_message() ! Get the completion status of the open call ! success returns rtr_mt_opened send_loop: rtr_send_to_server( ...RTR_F_SEN_ACCEPT....) ! Send a tx-message and ! implicitly start a new tx rcv_loop: rtr_receive_message() ! Find out what RTR wants to tell us next switch (message_received_type) { case rtr_mt_reply: Process_Reply_from_Server; break; case rtr_mt_rtr_event: Process_RTR_Event; break; case rtr_mt_user_event: Process_User_Event; break; case rtr_mt_accepted: Tell_User_It_Worked; break; case rtr_mt_rejected: Tell_User_About_Failure; break; } IF ( message_received_type = rtr_mt_accepted ) OR ( message_received_type = rtr_mt_rejected ) THEN GOTO send_loop ! Last transaction done, issue the next one ELSE GOTO rcv_loop ! Get the next incoming message

In Example 2-1, note that the switch statement tests on message type. All messages that are received from RTR have a message type; for further information, see Section 2.10, RTR Messages.

2.5.2 Simple Server

Example 2-2 is a simple server that receives transactions and events.

Example 2-2 Example Server

rtr_open_channel() ! open a channel to the desired facility rtr_receive_message() ! get the completion status of the open call ! success returns rtr_mt_opened rcv_loop: rtr_receive_message() ! Find out what RTR wants to tell us next CASE message_received_type OF rtr_mt_msg1: Do_Some_SQL_And_Maybe_Send_A_Reply; rtr_mt_msgn: Do_Some_More_SQL_And_Maybe_Send_A_Reply; rtr_mt_prepare: Accept_or_Reject_Tx ; rtr_mt_rtr_event: Process_RTR_Event; rtr_mt_user_event: Process_User_Event; rtr_mt_accepted: Commit_DB ; rtr_mt_rejected: Rollback_DB ; END_CASE; GOTO rcv_loop

2.6 Using the Portable API

As can be seen from the examples in the previous section, an application first opens one or more channels by calling rtr_open_channel .

The application can then process transactions and events on the channels it has opened. When a channel is no longer needed, the application closes it by calling rtr_close_channel .

A transaction becomes associated with a channel in one of the following circumstances:

1. When a client issues the first rtr_send_to_server call on a previously idle channel.
2. When a server receives from a client the first message belonging to a transaction by calling rtr_receive_message .
3. When a client issues a rtr_start_tx call on a previously idle channel.

From this point on the channel remains associated with the transaction until one of the following occurs:

1. The application rejects the transaction using rtr_reject_tx .
   • The transaction is over, no more messages will be received on behalf of this transaction.
   • The channel becomes idle, ready for initiation/reception of another transaction.
2. The application accepts the transaction using rtr_accept_tx
   • After calling rtr_accept_tx the application may continue receiving messages belonging to the transaction. However, it cannot subsequently either reverse its decision to accept by calling rtr_reject_tx , or (in the case of a client application) make additional calls to rtr_send_to_server .
   • The final message received for a transaction will always be a transaction completion status; either rtr_mt_rejected or rtr_mt_accepted .
   • The channel becomes idle, ready for initiation or reception of another transaction.
3. The application receives, by a call to rtr_receive_message , a completion status indicating that the transaction has been rejected by some other participant.
   • The transaction is over. No more messages will be received, and no more calls may be made on behalf of this transaction.
   • The channel becomes idle, ready for initiation or reception of another transaction.

Note that RTR considers a transaction to have been committed to the database (so that it does not need to replay it in case of failure) when the server indicates willingness to receive a new transaction by calling rtr_receive_message on the channel, after having received the transaction completion status.

Using RTR_ANYCHAN for the prcvchan argument implies that the transactions are committed on all channels, and calling rtr_receive_message on a single channel implies that the transaction on that channel is committed.

Calling rtr_close_channel also indicates to RTR that the last transaction has been committed.

2.7 Concurrency

The routine rtr_receive_message is used by an application to receive all incoming messages, responses and events. This provides a single consistent method of information delivery.

All RTR routines other than rtr_receive_message complete immediately, and any responses are queued for later reception by rtr_receive_message .

The application calling rtr_receive_message may choose whether (and how long) it should wait for an incoming message to arrive (if there is no message available for immediate reception).

In addition, the application may optionally specify a "wakeup routine" to be called by RTR when a message becomes available for reception.

2.8 Using the RTR Set Wakeup Routine

An application program may typically wish to respond to input from more than one source. An example of this is an application program which prompts for user input in a window and at the same time displays information received asynchronously via broadcast events.

In order to avoid the application polling its various input sources, RTR provides the rtr_set_wakeup routine. This allows the application to specify a routine to be called when there is data to be received from RTR but no call to rtr_receive_message is currently active. The application program can then be coded as follows:

#include <stdlib.h> void app_wakeup_routine (void) { /* NB This is called from an AST, ALRM or IO signal handler, * or another thread depending on the platform. * Although RTR blocks signals, ASTs and the wakeup thread * until it is safe and convenient, * you may prefer to just set a flag or generate an event and * perform the receive_message in your main thread instead. */ /* Get all outstanding rtr messages */ do { sts = rtr_receive_message(..., /* timoutms */ 0 ) ; check ( sts ) ; process_message () ; } while ( sts != RTR_STS_TIMOUT ) ; } static void app_cancel_wakeup (void) { rtr_set_wakeup( NULL ); } main () { sts = rtr_set_wakeup( app_wakeup_routine ); atexit(app_cancel_wakeup); . . }

If RTR data is available when rtr_set_wakeup is called, the application's wakeup routine is called immediately.

Note that the wakeup handler itself is not permitted to call any function that might have to wait; the only RTR call allowed in the wakeup handler is rtr_receive_message called with a zero timeout.

2.9 API Optimizations

Reliable Transaction Router introduces greater flexibility and efficiency in how transactions are packaged at the client.

The total sequence of events that a client application has to execute are as follows:

1. Start a transaction.
2. Send one or more transaction messages, optionally receive one or more transaction messages.
3. Either accept or reject the transaction.
4. Wait for the transaction accept or reject message and process accordingly.
5. Return to Step 1.

In Reliable Transaction Router, all these steps can be used if required, but optimizations allow some of the steps to be handled implicitly.

• The call to rtr_start_tx (Step 1) may be omitted if, for example, no timeout is required for the transaction. A call to rtr_send_to_server on a channel that does not have an active transaction automatically implies a call to rtr_start_tx .
• Step three may be handled implicitly if the client wishes to accept the transaction. This is done by setting the RTR_F_SEN_ACCEPT flag on the last (or only) call to rtr_send_to_server .