Inside a large data center, most traffic never leaves the building: it runs from one rack of servers to another. The networks built for that look nothing like the tree-shaped office networks they replaced. They are grids of identical switches in which every rack is two hops from every other, every path is used at once, BGP runs on every cable, and the tenants' own networks ride on top in tunnels. This chapter takes that design apart: the geometry, the arithmetic, the routing, and the overlay.
Common mix-up: the fabric and the overlay are two different networks. The fabric (the "underlay") is plain IP routing between switches; it only knows switch loopbacks and link addresses. The tenants' VLANs, MAC addresses and private subnets exist only in the overlay, carried inside VXLAN tunnels between the leaf switches. EVPN is the overlay's control plane; it does not route the fabric.
This chapter builds on switching and tunnelling. If VLANs, MAC learning and ARP are new, start with Ethernet on networks.jelia.nyc; for what encapsulation does to a packet and its MTU, read VPNs and overlays. For equal-cost multipath and flow hashing, chapter 03 is the reference. Everything here assumes those.
The classic enterprise data center was a tree in three tiers. Servers plugged into access switches at the top of each rack; access switches uplinked to a pair of aggregation switches; aggregation pairs uplinked to a pair of big core routers at the root. It was designed for "north-south" traffic: requests coming in from users outside, answers going back out. Most of the bandwidth was needed near the edge, so the tree narrowed toward the top.
Three things broke it. First, traffic turned sideways. A single web request now fans out to dozens of internal services, databases, caches and storage nodes, so most bytes travel "east-west", rack to rack, and every one of those bytes had to climb the tree and come back down through the narrow top. Second, the tree was a Layer 2 network, and Layer 2 networks must not contain loops, because an Ethernet frame has no hop count and would circle forever. The Spanning Tree Protocol prevents loops by switching redundant links off. RFC 7348 puts it bluntly: with STP, operators are "effectively paying for more ports and links than they can really use," and "resiliency due to multipathing is not available." Third, a Layer 2 domain is a single failure domain: one broadcast storm, one flapping link, one confused switch, and the whole tree feels it.
The replacement is a leaf-spine fabric. Every rack gets a leaf switch. A row of spine switches sits above. Every leaf connects to every spine, and nothing else connects to anything: leaves never link to leaves, spines never link to spines. Any server can reach any other server in at most leaf, spine, leaf. Because the fabric runs IP routing rather than spanning tree, no link is blocked, and with as many spines as you like, there are that many equal paths between any two racks.
In 1953 Charles Clos, an engineer at Bell Labs, published "A Study of Non-Blocking Switching Networks" in the Bell System Technical Journal. Telephone exchanges needed to connect any caller to any free line without building one enormous crossbar. Clos showed how to build a big switch out of three stages of small ones (ingress, middle, egress) and worked out how many middle-stage switches you need so that a new call can always be connected without disturbing existing ones.
A leaf-spine fabric is that same three-stage network folded in half: each leaf plays both the ingress and the egress stage, and the spines are the middle stage. That is why engineers call a two-layer leaf-spine fabric a "3-stage Clos" and a three-layer one a "5-stage Clos": count the switches a packet crosses on the longest path, not the layers on the drawing. Al-Fares, Loukissas and Vahdat's 2008 SIGCOMM paper showed the same arithmetic could build a full-bandwidth network for 27,648 hosts out of nothing but identical 48-port commodity switches, which helped turn the idea into common practice.
A fabric's size is set almost entirely by one number: the radix of its switches, meaning how many ports each one has. Everything else follows by counting.
Take a leaf with k ports. Some of them go up to spines, the rest go down to servers. Call them u uplinks and d downlinks, so u + d = k. If each leaf runs one cable to each spine, the number of spines equals u. Every spine port is a cable to a different leaf, so a spine with ks ports can serve at most ks leaves. That is the ceiling of a 3-stage fabric: leaves ≤ spine radix.
The second number is the oversubscription ratio: how much capacity faces the servers on a leaf, compared with how much faces the spines.
The third is bisection bandwidth. Cut the fabric down the middle, half the racks on each side, and ask how much traffic can cross the cut. In a leaf-spine, every byte crossing between halves goes up some leaf's uplinks, so the answer is half the leaves times each leaf's usable uplink capacity, capped by what its servers can actually send. Al-Fares and colleagues define oversubscription the same way: the ratio of the worst-case bandwidth available to hosts to the total bisection bandwidth.
Worked example. A common leaf shape has 48 ports at 25 Gb/s for servers and 8 ports at 100 Gb/s for uplinks: 56 ports. With 8 spines of 64 ports each, one cable per leaf per spine:
To grow past 64 racks you either buy bigger spines or add a layer. A 5-stage fabric builds several leaf-spine "pods" and joins their spines through a layer of super-spines. With identical switches of radix k and no oversubscription, a 3-stage fabric tops out at k²/2 servers and a 5-stage one at k³/4: for 48-port switches, 1,152 against 27,648. Facebook's 2014 fabric is a well-documented 5-stage design: pods of 48 server racks, each served by four fabric switches, joined through four independent planes of spine switches, each plane scalable to 48 devices.
For a non-blocking 3-stage fabric with radix-k switches, split each leaf's ports evenly: k/2 down, k/2 up. That needs k/2 spines, and each spine's k ports serve k leaves. Servers = k leaves × k/2 = k²/2.
For five stages, make each pod a non-blocking 3-stage block in which the "spines" (now called fabric or aggregation switches) also split their ports half down, half up. A pod then has k/2 leaves and k/2 fabric switches, giving (k/2)² servers and (k/2)² uplinks. The super-spine layer has (k/2)² switches of k ports, so it can serve k pods. Servers = k × (k/2)² = k³/4. With k = 48 that is 27,648, the figure in the Al-Fares paper.
Real fabrics rarely hit these ceilings exactly. They oversubscribe the leaves, use faster uplinks than downlinks (one 400 Gb/s port standing in for several slower ones), or break a big port into four smaller ones with a breakout cable, all of which change the counting. The instrument below lets you vary each of these.
Set the switches and the speeds. The calculator works out how many servers the fabric holds, its oversubscription, bisection bandwidth and cable count, and draws it. Past about 24 leaves the drawing shows the first and last few and summarizes the middle, because 64 leaves and 512 cables would be a grey smear.
A fabric of routed links needs a routing protocol. The obvious choice would be an interior protocol such as OSPF or IS-IS (chapter 05, chapter 06). Many large operators chose BGP instead, the protocol built to carry policy between independent networks on the internet, and run nothing else. RFC 7938, "Use of BGP for Routing in Large-Scale Data Centers," published in 2016 with authors from Facebook and Arista Networks, describes the design.
Its rules are short:
That AS plan is quietly clever. A route from leaf 6 reaches leaf 1 with the AS path "spine-AS, leaf-6-AS" through every spine. The paths are identical in length, so BGP's best-path process (chapter 09) ties, and with multipath enabled the leaf installs one next hop per spine: ECMP for free. And because all spines share one AS, a route can never go leaf, spine, leaf, spine: the second spine would see its own AS in the path and reject it as a loop. BGP's ordinary loop prevention keeps traffic on the shortest valley-free paths without any extra configuration.
Why BGP rather than an IGP? RFC 7938 gives the reasons. BGP's state machine is simpler, it leans on TCP for reliable delivery, and a failure is masked rather than flooded: when a link dies, each BGP speaker just picks its next best path and sends on only what changed, while a link-state protocol floods news of the failure through the entire area and every router reruns its shortest-path calculation. BGP also gives the operator per-session policy at every hop, for draining a switch before maintenance or steering traffic. The objections the RFC lists, that BGP is "WAN only," slow to converge and fiddly to configure, are answered by the design itself: with one-hop sessions and directly connected next hops, failures are detected by the link going down (or by BFD, chapter 07) rather than by BGP's slow hold timer.
| Switch | AS (example) | Sessions | What it advertises |
|---|---|---|---|
| Spine 1 to 4 | 65100 (shared) | one per leaf | Every leaf's prefixes, passed on unchanged except for its AS prepended. |
| Leaf 1 | 65001 | one per spine | Its rack subnet, e.g. 198.51.100.0/27, and its VTEP loopback 192.0.2.11/32. |
| Leaf 2 | 65002 | one per spine | 198.51.100.32/27 and 192.0.2.12/32. |
| Leaf 6 | 65006 | one per spine | 198.51.100.160/27 and 192.0.2.16/32. |
Addresses from the RFC 5737 documentation blocks; AS numbers from the private-use range of RFC 6996, as RFC 7938 recommends.
The original 2-byte private range holds only 1,023 numbers, and a large fabric can have more leaves than that. RFC 7938 suggests reusing the same leaf AS numbers in every pod and configuring "allowas-in" on the leaves, so a leaf will accept a route that carries its own AS number from a different pod. The RFC notes this is not standardized but widely implemented. The cleaner fix is the 4-byte private range from RFC 6996, 4200000000 to 4294967294, which has room for about 95 million devices.
When the same prefix is announced from two leaves (an anycast service, or a server dual-homed to two racks), the paths arriving at a third leaf differ in their AS numbers even though they are the same length. Standard BGP multipath requires identical AS paths, so RFC 7938 calls for the feature vendors name "multipath relax" or "multipath multiple-AS," which treats equal-length paths through different AS numbers as equal.
Numbering every fabric link is tedious: 512 cables need 512 tiny subnets, and both ends must agree. Most fabric software can instead run each session over the link's IPv6 link-local address, which every interface creates for itself, and discover the neighbor from IPv6 router advertisements. FRRouting calls this an "interface-based (unnumbered)" peer: you name the interface, not an address, and FRRouting notes it is "useful in data center environments where point-to-point (unnumbered) links" are used.
That leaves a puzzle: how do you advertise an IPv4 rack subnet over a session that has no IPv4 address, with no IPv4 next hop to give? RFC 8950 (which replaced RFC 5549) lets BGP advertise IPv4 prefixes with an IPv6 next hop. The leaf installs the IPv4 route pointing at the neighbor's link-local IPv6 address, and the forwarding plane resolves that to the neighbor's MAC address. No IPv4 addresses on fabric links at all, only on loopbacks and server-facing interfaces.
With four spines, a leaf has four equal next hops for every remote rack. Chapter 03 explains how a router spreads flows across equal next hops: it hashes each packet's 5-tuple (addresses, protocol, ports) and uses the result to pick a path, so every packet of one connection takes the same path and stays in order. A fabric does exactly that, at every leaf and every spine, millions of times a second.
Hashing spreads flows, not bytes. That is fine when traffic is many small conversations, the "mice" that make up most connections. It goes wrong with elephant flows: a few long, huge transfers such as a storage replication or a backup. Two elephants that happen to hash to the same spine share one uplink while another spine's uplink sits idle, and the hash has no idea. Fabrics fight this with more and faster uplinks, so that one elephant is a smaller share of any link, and by keeping oversubscription low on the racks that move the most data.
The payoff of so many equal paths is a small blast radius. When one of four spines fails, each leaf loses one next hop out of four. Every pair of racks keeps three-quarters of its capacity, and only the flows that were using the dead spine need a new path. Compare a spanning tree: the root switch fails, the tree must be rebuilt from scratch, and every flow in the domain stops until it is. The instrument below lets you break things both ways and count the damage.
Click or tap a spine to fail it, or a cable to cut it (tap again to repair). The grid shows the capacity left between every pair of leaves. The simulator also sends 3,000 flows between random racks and counts how many had to change path: the unavoidable minimum (flows whose spine is gone) and the number a naive "hash modulo N" router would reshuffle. Switch to Tree + STP to run the same switches as a Layer 2 network: spanning tree keeps only a tree of links, rooted at the lowest-numbered live spine, and blocks the rest.
Every cable is 100 Gb/s. Capacity is per leaf pair with nothing else running; in a real fabric all pairs share the spines, so this is a ceiling, not a promise.
A routed fabric gives each rack its own subnet. But the virtual machines running on it often need something else: a customer's private network, with its own addresses, that may stretch across many racks, and that must stay invisible to every other customer. VLANs used to do that job, but a VLAN tag has only 12 bits, so 4,094 usable segments, and VLANs live in Layer 2, the thing the fabric just got rid of.
VXLAN, described in RFC 7348, solves both. A leaf switch (or a hypervisor) acting as a VTEP, a VXLAN tunnel endpoint, takes the tenant's whole Ethernet frame, puts an 8-byte VXLAN header in front of it, and sends it inside an ordinary UDP/IP packet to the VTEP where the destination lives. The fabric only sees IP packets from one leaf's loopback to another's and routes them like any other traffic. The header's 24-bit VNI (VXLAN Network Identifier) says which tenant segment the frame belongs to: 16,777,216 possibilities.
| Layer (outermost first) | Bytes | What it carries |
|---|---|---|
| Outer Ethernet | 14 (18 with a VLAN tag) | One fabric hop: this switch's MAC to the next switch's MAC. Rewritten at every hop, like any routed packet. |
| Outer IPv4 | 20 (IPv6: 40) | Source VTEP to destination VTEP, e.g. 192.0.2.11 → 192.0.2.16. This is what the fabric routes. |
| Outer UDP | 8 | Destination port 4789. Source port: a hash of the inner frame's headers, recommended in 49152–65535, so different inner flows take different fabric paths. |
| VXLAN header | 8 | Flags (the I bit says "VNI is valid"), 24 reserved bits, the 24-bit VNI, 8 reserved bits. |
| Inner Ethernet + payload | 14 + tenant packet | The tenant's original frame, untouched. RFC 7348 says the VTEP should strip any inner VLAN tag unless configured otherwise. |
The UDP source port is the trick that makes VXLAN fabric-friendly. Every tunnel between two leaves has the same outer addresses, so a fabric hashing only the outer headers would put all of one tunnel's traffic on one spine. Because the encapsulating VTEP fills the source port with a hash of the inner packet, each tenant flow gets its own outer 5-tuple and its own spread across the spines.
The cost is size. Over IPv4 the tunnel adds 50 bytes to each frame, and RFC 7348 forbids the VTEP from fragmenting: oversized VXLAN packets may simply be dropped. So the fabric's MTU must be raised to hold the tenant's packet plus the inner Ethernet header plus 36 bytes of outer IPv4, UDP and VXLAN, or the tenant's MTU must shrink. That is why fabric links are commonly configured for jumbo frames, well above 1,500 bytes.
How does a VTEP know which remote VTEP holds a given MAC address? RFC 7348's original answer was flood and learn, borrowed from switches: map each VNI to an IP multicast group, flood unknown and broadcast frames to every VTEP in the group, and learn "this MAC lives behind that VTEP" from the source of arriving packets. It works, but it needs multicast routing in the fabric (chapter 13), it floods, and a VTEP learns nothing until traffic flows.
EVPN replaces flooding with announcements. Defined for MPLS networks in RFC 7432 and adapted for VXLAN in RFC 8365, it adds a new address family to BGP (AFI 25, L2VPN; SAFI 70, EVPN) whose "prefixes" describe Layer 2 facts. When a leaf learns that a virtual machine with MAC 00:00:5e:00:53:20 and IP 198.51.100.20 sits on one of its ports in VNI 10100, it announces that over BGP, and the spines pass it to every other leaf. Each leaf now knows exactly which VTEP to tunnel to before the first packet is sent. The same eBGP sessions that build the underlay can carry it, or a separate set of sessions between loopbacks.
Five route types do most of the work. In plain words:
| Type | Name | What it says, in plain words |
|---|---|---|
| 1 | Ethernet auto-discovery | "I am attached to this multi-homed server link." Used for fast failover and load sharing when one server connects to two leaves. |
| 2 | MAC/IP advertisement | "MAC 00:00:5e:00:53:20, with IP 198.51.100.20, is behind me, in this VNI." The workhorse: replaces MAC learning and, with the IP included, lets leaves answer ARP locally. |
| 3 | Inclusive multicast Ethernet tag | "I have hosts in VNI 10100; send me that segment's broadcast and unknown traffic." Builds the flood list, usually by ingress replication: the sending leaf copies the frame to each listed VTEP, so no fabric multicast is needed. |
| 4 | Ethernet segment | "I share this multi-homed link with these other leaves." Used to elect which of them forwards broadcast traffic to it, so the server does not get two copies. |
| 5 | IP prefix (RFC 9136) | "Subnet 203.0.113.0/24 of this tenant is reachable through me." Routes whole prefixes, not single hosts, for routing between tenant subnets and to the outside. |
Two more ideas complete the picture. The first is the anycast gateway. Every leaf that serves a tenant subnet carries that subnet's default gateway, with the same IP and MAC address on every leaf (RFC 9135). A virtual machine always finds its gateway one hop away, on its own leaf, and if it is moved to another rack it keeps the same gateway address and MAC without noticing. Routed traffic between tenant subnets leaves the rack from the first leaf instead of hair-pinning through a central router.
The second is how a frame actually gets routed between two tenant subnets on different leaves, which RFC 9135 calls integrated routing and bridging (IRB). It comes in two flavours, described in the panel below.
In asymmetric IRB, the ingress leaf does all the work: it routes the packet from the source subnet into the destination subnet, then bridges it in the destination subnet's VNI straight to the egress leaf, which only bridges. That means every leaf must carry every subnet's VNI and every host's ARP entry, even for tenants it does not serve.
In symmetric IRB, both leaves route. The ingress leaf routes the packet into a per-tenant Layer 3 VNI, a transit segment that stands for the tenant's whole routing table (its VRF), and tunnels it to the egress leaf; the egress leaf routes it again from the L3 VNI into the destination subnet. Each leaf only needs the subnets it actually hosts, plus host routes (type 2 with IP) and prefix routes (type 5) learned over EVPN. RFC 9135 notes that in symmetric IRB a leaf only maintains ARP entries for locally attached hosts. That scales better, which is why it is the usual choice in large fabrics.
The route type 5 in instrument 3 carries the L3 VNI in its label field; RFC 8365 likewise carries the VNI in the label field of a type 2 route, which in MPLS EVPN would hold an MPLS label.
Choose the tenant's packet, the VNI and the tunnel endpoints. The builder stacks the headers with real byte counts, writes out the 8-byte VXLAN header, computes the hashed UDP source port, tells you the smallest underlay MTU that carries it without being dropped, and shows the EVPN route a leaf would send for the event you pick.
Headers drawn to scale against each other; the tenant's frame is drawn at a fixed width with its real size written on it. Dashed = the tenant's own frame, carried untouched.
| Layer | Bytes | Fields |
|---|
Follow one packet through everything in this chapter. A virtual machine on leaf 1, address 198.51.100.10 in tenant VNI 10100, sends to 198.51.100.20 in the same subnet, which lives on leaf 6.
Neither spine ever looked inside. The spines hold only underlay routes, a few hundred loopbacks and rack subnets, while all the tenant state lives at the edge, in the leaves that serve those tenants. That division is why a fabric can host thousands of tenants on spines that know nothing about any of them.
MTU mismatches are the classic fabric outage: small packets work, large ones vanish, and pings succeed while file transfers hang. The fix is a consistent jumbo MTU on every fabric link and a test that sends full-size packets with fragmentation disallowed.
Uneven hashing shows up as one spine's uplinks running hot while the others idle; it usually means elephant flows or a hash that ignores the VXLAN source port. Chapter 03 covers polarization and resilient hashing.
Draining before maintenance is a routing operation: an operator makes a spine announce its routes with a longer AS path (or withdraw them), leaves shift their traffic to the other spines, and only then is the switch touched. RFC 7938 describes this kind of traffic engineering with BGP attributes. Chapter 15 covers how to see what the routing system is actually doing.