Clos, CBB, and Fat Tree Networks

The concept of a CBB or Fat-Tree topology is based on the seminal 1953 work on non-blocking switch networks by Charles Clos. In general, the Clos, CBB, and Fat Tree networks describe identical switch topologies, however, in high performance compute clusters the CBB nomenclature is most common and thus will be used from here on.

So what is the point of building a CBB network? Given a basic switch building block with N ports and the requirement to build a switch network to connect to M end nodes (where M > N), a CBB network defines an efficient topology (lowest latency) using multiple levels of basic switch elements providing a non-blocking or blocking switch configuration supporting M endpoints. In such multi-level switch architectures, some of the ports of each basic switch element are connected to end nodes, while other ports are connected internally to additional levels of basic switch elements which do not connect to endpoints.

The key attribute of the non-blocking CBB topology is that for any given level of the switch topology, the amount of bandwidth connected to the downstream end nodes is identical to the amount of bandwidth connected to the upstream path used for interconnection. Note that the links are bidirectional so the notion of upstream and downstream describes the direction of the interconnect topology towards or away from the shortest path to an end node, rather than the actual data flow.

In Blocking CBB, the amount of bandwidth connected to the downstream end nodes is higher than the amount of bandwidth connected to the upstream path used for interconnection. In a 2:1 blocking scenario, the amount of bandwidth connected to the downstream end nodes is twice the amount of bandwidth connected to the upstream path used for interconnection. Blocking topologies reduce the bandwidth capability in the networks and can be used for applications that do not require high bandwidth. In most cases, non-blocking CBB is the preferred solution.

Two Level CBB Network Example

Consider the two level switch topology shown in Figure 2. It is constructed with basic switch building blocks having P ports. Suppose there are N1 Level 1 switches, each of which has P/2 ports connected downstream to end nodes, and an identical number connected upstream to the second level switch. The total number of end points connected is then: N1 * P/2. Suppose for example each basic switch building block has eight ports and a two level network is constructed with four ports connected both to end points and Level 2 switches as shown. There are four Level 1 switches so: P=8 and N1=4, hence using an 8 port building block this topology builds a 16 port non-blocking switch. It is possible to connect more total endpoints by increasing the number of Level 1 switches and even further by increasing the number of levels in the switch topology. The key to making the configurations non-blocking is to always preserve identical bandwidth (upstream and downstream) between any two levels of the multi-level switch topology.