Deadlock-free dynamic reconfiguration over InfiniBand™ NETWORKS

Abstract

InfiniBand Architecture (IBA) is a newly established general-purpose interconnect standard applicable to local area, system area and storage area networking and I/O. Networks based on this standard should be capable of tolerating topological changes due to resource failures, link/switch activations, and/or hot swapping of components. In order to maintain connectivity, the network's routing function may need to be reconfigured on each topological change. Although the architecture has various mechanisms useful for configuring the network, no strategy or procedure is specified for ensuring deadlock freedom during dynamic network reconfiguration. In this paper, a method for applying the Double Scheme over InfiniBand networks is proposed. The Double Scheme provides a systematic way of reconfiguring a network dynamically while ensuring freedom from deadlocks. We show how features and mechanisms available in IBA for other purposes can also be used to implement dynamic network reconfiguration based on the Double Scheme. We also propose new mechanisms that may be considered in future versions of the IBA specification for making dynamic reconfiguration and other subnet management operations more efficient.

Keywords:

• Deadlock-free dynamic reconfiguration
• InfiniBand architecture
• Double scheme
• Network management

Acknowledgements

We acknowledge the efforts of Rafael Casado and Francisco J. Quiles in helping to provide support for IBA modeling and simulation on OPNET™, and we are grateful to OPNET for use of their OPNET Modeler simulation tool. This research is supported partly by NSF grant CCR-0209234 and Spanish CICYT grant TIC2003-08154-C6.

Notes

Deadlock can occur when packets block cyclically waiting for resources while holding onto other resources indefinitely [9,10]. If allowed to persist, deadlocks can bring the entire system to a standstill, making it vitally important for both the routing algorithm and the reconfiguration technique to guard against them.

IBA specifications require that any data packets in the management virtual lane be dropped.

It is expected that this would not cause any significant QoS degradation as most implementations are likely not to use all sixteen SLs at once.

IBA specifications call the vendor-specific management packets “General Management Packets”, as the term “SubnetManagement Packet” is restricted to the subnet management class, as defined in the specifications. Similarly, management methods such as Get( ) and Set( ) for vendor-specific attributes are called VendorGet( ) and VendorSet( ), as opposed to SubnGet( ) and SubnSet( ).

According to IBA specifications, a port in INITIALIZE state accepts management packets but not data packets.

This includes SMPs and vendor-specific GMPs used for drainage. In this section, we will use the term SMPs for both subnet and general management packets.

*At application data injection rate of 145,000 packets/node/s that we used in this experiment, the increase in the number of management packets from 388 for static to 716 for Double Scheme corresponds to less than 0.04% of the total network load.