10    Flow Control

The DIGITAL UNIX ATM subsystem uses the following types of flow control:

• Hardware flow control -- The ATM network uses this to manage congestion.

• Software flow control -- The DIGITAL UNIX ATM subsystem uses this to control the movement of data between modules.

Although these methods of flow control are different, they do interact, especially in the transmission of data on the network. You must design and implement each device driver and convergence module to use flow control properly, thus ensuring the proper operation of the entire subsystem.

10.1    Hardware Flow Control

The ATM network and adapter use hardware flow control to control congestion within the network. This allows the network to prevent the sender from transmitting more data than the network can handle. In the UNI 3.0/3.1 environment, there is no standard for ATM flow control, although many vendors implement their own methods of flow control. Since these flow-control methods are confined to the ATM adapter and since the DIGITAL UNIX ATM subsystem accommodates as many types of adapters as possible, all methods of flow control are accommodated as long as they are isolated to the adapter and its device driver. The ATM subsystem also supports the use of both proprietary and standard methods of hardware flow control.

To enable hardware flow control, you must send a device driver the ATM_DRVMGMT_FC command through the driver's management function. The argument to this command indicates whether to enable flow control (and the type to enable) or to disable all flow control. Drivers must be initialized with flow control disabled. Currently, the CMM recognizes the following types of hardware flow control:

• Vendor-specific flow control

  Requires that the adapter and the switch implement the same type of hardware flow control. This command is sent as the result of the atmconfig +vfc command being issued by an operator (the system is unable to determine which types of vendor-specific protocols are usable since this requires knowledge of the protocols that both the adapter and the switch implement).

• ATM Forum standard flow control

  Currently not supported. Generally, the CMM uses the fields in the switch and driver Management Information Bases (MIBs) to detect flow control, and turns on this flow control without any user interaction. However, system administrators can also enable this form of flow control by running the atmconfig +sfc command.

To disable flow control, a system administrator uses either the atmconfig -sfc or atmconfig -vfc command, depending on the type of flow control currently enabled.

10.2    Software Flow Control

If you do not use hardware flow control, the driver might be passed data from convergence modules faster than it can transmit the data on the interface. In this case, the driver must be able to inform the sending convergence modules that its queues are full and that no data should be sent until further notification. Rather than having the driver perform an explicit notification (which involves extra function call overhead) to do this, the driver must return a value from its transmit function indicating that its queue is full (or is filling). The driver should then consider the queue flow controlled and must send a notification to the CMM when the queue is ready to accept more data.

In its implementation, the CMM records most information associated with flow control. Since the driver does all software flow control on a per-queue basis, the CMM must translate this into information that can be used on a per-VC basis. The CMM maintains a list of which VCs are assigned to which driver queues; the CMM converts driver queue enable notifications to per-VC enable notifications for convergence modules.

Drivers must maintain the following values on their queues: a high-water mark and a low-water mark. Drivers use these marks to control the value returned by the transmit function and the sending of flow-control notifications to the CMM.

10.2.1    High-Water Mark

The high-water mark controls when the driver starts returning warnings (ATM_CAUSE_QWARN) from the transmit call. These warnings are used by convergence modules that might try to start queuing before the driver can no longer accept data. The use of warnings also allows convergence modules the chance to use flow control on the upper-layer protocol modules, if possible, before the driver actually starts rejecting data. The driver should start returning the queue warning when the number of queued messages passes its high-water mark. The driver writer must choose a high-water mark that is appropriate for the driver's queuing policy and adapter characteristics.

When a driver returns a queue warning, it must have accepted the data for transmission and queued the data. The warning returned does not indicate that the driver cannot accept the data, only that the queue is close to being full.

When a driver's queue is full, the driver must not discard data passed in the transmit function. Instead, the driver must return ATM_CAUSE_QFULL from the transmit function so the convergence module can decide if the data is to be dropped or queued within the convergence module. The queue full return indicates that the driver cannot accept the data for transmission because its queue is full, not for any other reason, such as insufficient memory. Drivers can continue to receive and reject packets for transmission even after the queue fill return, since more than one convergence module could be queuing to the queue.

10.2.2    Low-Water Mark

When the number of messages on a driver's transmit queue drops below the low-water mark, the driver sends an ATM_DE_STARTQ notification to the CMM. The driver must send this notice if it has returned any queue warning or full conditions from its transmit function since the last restart notification was sent. This implies that drivers maintain state information about previous returns and notifications. Driver writers can provide extra restart notifications (the CMM properly handles redundant notification), but should not provide too few notifications. The driver notifies restarts per queue only. The CMM converts these to per-VC restarts and sends those notifications to convergence modules. Therefore, drivers need to maintain only the per-queue state, not the per-VC state, for flow control.

10.3    Convergence Module Flow Control

Convergence modules are not required to maintain any type of transmit queue since ATM device drivers must provide some amount of output queuing. However, convergence modules should provide transmit queues if data not accepted for transmission by a driver is to be saved for retransmission. If it is acceptable to discard data, convergence modules can ignore queuing and flow-control features all together.

Device drivers generate flow control on a per-queue basis; convergence modules handle flow control on a per-VC basis. The reasons for this are as follows:

• Information concerning driver queuing policies and CMM assignment of VCs to driver queues is not available to convergence modules.

• The CMM uses flow control on the sender synchronously so that only those VCs trying to send data to a flow-controlled queue actually receive flow-control indications. Those VCs that do not send any data to the affected queue while the flow-control condition exists never receive a flow-control indication and are unaffected by the flow-control condition.

• Only those VCs that have been flow controlled are notified when they can resume transmission of data.

If a convergence module implements transmit queuing, it must be prepared to handle software flow control from the CMM. Convergence modules are given flow-control information in the return from the CMM send routine and by an exception notification. The convergence module should use this information to control its queuing and sending of data.

Convergence modules can bypass internal queues for efficiency as long as data is flowing freely to the CMM. A convergence module needs to use transmit queues only when it is informed that the device driver's queue is filling up or is full. Once the queues have drained, the queues can be bypassed again. This type of implementation requires some extra state to be maintained in the convergence module. However, this implementation could increase the speed of the nonqueued transmit path.

When a convergence module calls atm_cmm_send, it receives an atm_err_t return value, indicating the disposition of the data by the CMM and device driver. If the device driver accepts the data for transmission, atm_cmm_send returns ATM_CAUSE_GOOD. In this case, the driver has queued the data for transmission and the convergence module must destroy any references to the data; the driver frees the mbufs associated with the data.

When a device driver's transmit queue fills up beyond the high-water mark, the driver starts returning ATM_CAUSE_QWARN to the CMM, which returns this value to the convergence module. When a convergence module gets an ATM_CAUSE_QWARN back from atm_cmm_send, this means that the data was accepted and queued by the driver, but that the convergence module should not send any more data. The driver's queue could fill up and it would have to start rejecting data. In this case, the convergence module should either start queuing outgoing data or start dropping subsequent packets until it receives notification that the driver can accept data again.

The convergence module might also notify the upper-layer protocols (if the flow can be controlled) and continue to send data to the driver until the upper layers stop sending. A queue warning might not be returned before a queue is completely full and the driver can no longer accept data. This is true in cases in which multiple VCs are feeding a single queue. The queue warning is intended to give the convergence module warning of an impending queue full condition so that it can take any appropriate action before the driver stops accepting data.

If a device driver's queue becomes completely full and the driver is unable to queue any additional data, ATM_CAUSE_QFULL is returned from atm_cmm_send. In this case, the driver did not accept and queue the data and the convergence module must consider the VC blocked and unable to transmit any more data. The module must either queue the data internally or discard it. Any further attempts to transmit will probably return a queue full condition.

For those cases where multiple VCs feed a single queue and the convergence module can control the upper-layer protocols, the convergence module should implement a small transmit queue to handle any data that is sent by the upper layers between the flow-control notification and the actual cessation of data transmission. A small transmit queue ensures sufficient storage space to prevent data loss if a driver queue suddenly becomes full due to the accumulation of data from another VC.

When a queue warning or a queue full condition is reported to a convergence module, the VC that received the condition is considered flow controlled; no more data should be sent on the VC. When the driver queue drains below its low-water mark, the driver notifies the CMM that it is ready to start accepting data on that queue. The CMM then sends an ATM_CME_START_VC notification to each convergence module, specifying that the affected VCs can resume data transmission. One notification is sent for every VC that receives a flow-control indication from one or more atm_cmm_send calls.

When a convergence module receives the ATM_CME_START_VC exception, it should arrange to start sending data from its queues down to the driver and to remove flow control from the upper-layer protocols, if necessary. Note that convergence modules should not drain their queues as part of the exception function call on which the ATM_CME_START_VC exception is received as this could be on an interrupt stack. Instead, convergence modules should schedule a kernel thread or timeout to run and process the queued data.