A busy router rarely has just one way to reach a destination. A static route, OSPF, BGP and a directly connected interface may all claim to know the way, sometimes for overlapping prefixes, sometimes through next hops that are themselves only reachable through another route. This chapter is about how the router settles those arguments, and what it does when two answers are exactly as good: it splits the traffic, flow by flow, with a hash.
Common mix-up: administrative distance never beats a longer prefix. Distance only breaks ties between routes for the same prefix. A /24 learned by a protocol the router trusts least still wins over a /16 from the one it trusts most, because longest-prefix match is applied to every packet, at forwarding time, after the per-prefix contest is over. And distance is a local setting: it is never sent to neighbors.
If longest-prefix match, metrics and the "two defaults" trap are new to you, start with routing tables on networks.jelia.nyc, and for the families of routing protocols, dynamic routing. This chapter assumes both and asks a narrower question: when the router holds several candidate routes, which one ends up in the forwarding table, and which one does a given packet use?
The answer comes in two stages that happen at different times. The first stage happens in the routing information base (RIB, see chapter 01) whenever a route arrives or leaves. For each distinct prefix, the RIB holds a contest among every source that offered that exact prefix and installs one winner (or a set of equal winners) into the forwarding table, the FIB. The second stage happens for every packet: the FIB finds the longest prefix that covers the destination address (chapter 02 shows how it does that fast). Because the second stage only sees winners of the first, the effective order of questions for any one packet is:
There is a zeroth question hiding in front of all of them: is the route usable at all? A route whose next hop cannot be reached, or whose outgoing interface is down, is not a candidate. Neither is a route with distance 255, which by long convention means "never use this."
| Source | FRRouting default distance | What it says about trust |
|---|---|---|
| Connected | 0 | The router can see the network on its own interface; nothing is more certain. |
| Static | 1 | An operator typed it, so it beats any protocol unless given a higher distance on purpose. |
| eBGP | 20 | Routes from another autonomous system, learned directly from that neighbor. |
| EIGRP | 90 | Listed in FRRouting's table; the protocol is described in RFC 7868. |
| Babel | 100 | A loop-free distance-vector protocol (chapter 04). |
| OSPF | 110 | The link-state interior protocol of chapter 05. |
| IS-IS | 115 | The other link-state protocol, chapter 06. |
| RIP | 120 | Hop-count distance vector, the least trusted interior protocol. |
| iBGP | 200 | Routes relayed from inside your own AS; deliberately worse than any IGP. |
| (any) | 255 | Not installed in the forwarding plane at all. |
Values from the FRRouting zebra documentation, which says they were chosen to match other routing suites. They are common defaults, not a standard: vendors differ in places, and every value can be changed per route or per protocol.
The two BGP numbers look strange until you ask what could go wrong. A route learned by eBGP comes from the network that actually owns or carries the prefix, so it is trusted more than any interior protocol's idea of the same prefix. A route learned by iBGP is a copy of something another router in your own network heard. If iBGP outranked the IGP, a router could install an iBGP route for one of your own internal prefixes, pointing at a BGP next hop that is itself reached through that internal prefix: a loop through its own tail. Ranking iBGP below every IGP means the IGP always wins for anything it knows about, and iBGP only fills in the rest.
The same reasoning explains why a static route's default of 1 is so close to connected's 0: if an operator went to the trouble of typing a route, the router assumes they meant it. That assumption is what the floating static route, below, turns upside down.
RFC 1812, the 1995 requirements document for IPv4 routers, describes route choice as a series of pruning rules applied to a candidate set: Basic Match (drop routes that don't cover the destination), Longest Match, Weak TOS, Best Metric, and finally Vendor Policy, "sort of a catch-all to make up for the fact that the previously listed rules are often inadequate." Its suggested vendor policy is administrative preference: an integer from 0 to 255, zero most preferred, 254 least, and 255 meaning the route must never be used.
The RFC lists metric before preference because it compares metrics only within a single routing domain; real routers compare across protocols by preference first, because an OSPF cost of 20 and a RIP hop count of 2 are not the same kind of number and cannot be compared. The RFC itself warns that preferences are "not safe in that it can easily be misused to create routing loops": nothing makes two neighboring routers agree on them.
Here is one router's candidate list. Type a destination, then edit any row: switch it on or off, change its source, prefix, distance or metric, or untick "next hop OK" to pretend its next hop has become unreachable. The grid shows which test knocked each candidate out, and the numbered steps explain the decision in words. Changing the source fills in FRRouting's default distance, which you can then override.
Distance is usually something you leave alone. The one place almost every operator sets it by hand is the floating static route: a static route deliberately given a distance worse than the dynamic protocol that normally carries the prefix, so that it sits in the RIB unused until the dynamic route disappears.
Picture a branch office with a fast fibre link to headquarters, running OSPF, and a slower backup link. On the branch router you configure a static route for the headquarters prefix through the backup link, with distance 250. While OSPF is healthy, the RIB sees two offers for the same prefix: OSPF at 110 and the static at 250. OSPF wins and goes into the FIB. The static route stays in the RIB as a loser. When the fibre goes dark, OSPF's adjacency drops, its route is withdrawn, and the static route is now the only offer left. It "floats" up into the FIB within the time it takes OSPF to notice the failure. When OSPF comes back, it wins again and the static sinks back.
In FRRouting the distance is the optional last number on a static route; its documentation gives the range as 1 to 255 with a default of 1. Try it in the chooser: pick the "Floating static" scenario, then switch the OSPF row off.
| State | Offers for 203.0.113.0/24 | RIB winner | In the FIB |
|---|---|---|---|
| Normal | OSPF [110/20] via fibre · static [250/0] via backup | OSPF, distance 110 < 250 | via fibre |
| Fibre down | static [250/0] via backup | static, the only offer | via backup |
| Fibre back | OSPF [110/20] · static [250/0] | OSPF again | via fibre |
The trick depends on the dynamic route actually disappearing. If the fibre fails in a way the router cannot see, such as a media converter in the middle that keeps the local port lit, the interface stays up, OSPF waits out its dead interval (commonly 40 seconds on a LAN) before giving up, and traffic is blackholed until then. That is what BFD fixes, covered in chapter 07.
The static route itself must also stay valid. A static route whose next hop is an address is only usable while that address resolves through some other route; if the backup link's own interface goes down, the backup disappears too. Some operators also point the floating static at a more specific or less specific prefix than the dynamic route by mistake. Then distance never gets compared at all, and the longest prefix decides, which is usually not what they intended.
An OSPF route arrives with an answer the router can use directly: "send it out interface eth1 toward 198.51.100.1, a neighbor on that link." A BGP route does not. The NEXT_HOP attribute in a BGP update is just an IP address, and inside a large network it is usually the loopback address of some border router several hops away, such as 192.0.2.9. The router cannot put "192.0.2.9" in the FIB as an outgoing interface. It has to look up 192.0.2.9 in its own routing table, find (say) an OSPF route for 192.0.2.9/32 via 198.51.100.1 on eth1, and use that. This is recursive next-hop resolution, and the forwarding entry for the BGP prefix ends up pointing at the IGP's next hop.
RFC 4271, the BGP-4 specification, makes this a rule: BGP speakers should exclude unresolvable routes from their decision, and a route is resolvable only if the routing table holds a route matching its next hop that is not itself resolved through the route in question. The last clause rules out a route resolving through itself, directly or in a ring.
The useful consequence is speed. When the IGP route to 192.0.2.9 disappears because that border router died, every BGP route using it becomes unresolvable at once, and the router can stop using all of them without waiting for BGP's own hold timer, which is measured in tens of seconds or minutes, to expire. FRRouting's zebra calls this nexthop tracking: protocols register the next hops they care about and are told when the resolution changes. Modern routers go further and keep the FIB in two layers, prefixes pointing at a shared next-hop object, so that repointing one object moves all the prefixes behind it in one write. That idea, often called a hierarchical FIB, reappears in chapter 07.
Recursion is also why the phrase next-hop-self exists. When a border router learns a route from an external neighbor, the NEXT_HOP is that neighbor's address on the shared link, which interior routers may not have a route for. By default iBGP passes the attribute along unchanged, so those interior routers would find the route unresolvable and ignore it. Configuring next-hop-self on the border router makes it rewrite the next hop to its own loopback, which the IGP does carry. The details belong to chapter 08.
Suppose the RIB holds four routes: iBGP 203.0.113.0/24 next hop 192.0.2.9; OSPF 192.0.2.9/32 via 198.51.100.1; OSPF 192.0.2.0/24 via 198.51.100.5; and a static default 0.0.0.0/0 via 198.51.100.254.
RFC 4271 asks that only the longest matching route be considered when resolving, which is exactly the behavior that makes step 3 dangerous if you don't restrict it.
Sometimes the contest ends in a tie: two or more next hops for the same prefix, from the same protocol, with the same metric. In a well-built network that is not an accident. Data-center fabrics (chapter 14) are designed so that every leaf has equal-cost paths through every spine, and a core network with two parallel 100 Gb/s links wants to use both. When multipath is enabled, the router installs all the tied next hops as a group and spreads traffic across them. That is equal-cost multipath, ECMP.
The obvious way to spread traffic is to take turns, packet by packet. It is also the wrong way. Two paths are never exactly equal in delay: different queue depths, different cable lengths, a busier line card. Spray one TCP connection's packets across them and they arrive out of order. RFC 2991 spells out the cost: when three or more packets arrive ahead of a late one, TCP assumes loss and enters fast retransmit, resending data that was never lost and cutting its sending rate. So per-packet spraying can make a connection slower than if it had used one path.
Instead, routers spread flows. A flow is the set of packets that share the fields that stay constant for a connection, classically the 5-tuple: source address, destination address, protocol, source port and destination port. The router computes a hash of those fields and uses the hash to pick a next hop. Every packet of a connection carries the same five values, so every packet gets the same hash and the same path, and order is preserved. Different connections get different hashes and spread out. The balance is statistical: many small flows average out well; a few big ones do not.
The fields matter. Ports sit in the transport header, which a fragment after the first doesn't carry, and which an encrypted tunnel hides. Hashing only on the 3-tuple (addresses and protocol) or the 2-tuple (addresses only) avoids those problems but lumps every connection between the same two hosts onto one path. IPv6 offers a third option: the 20-bit flow label in the IPv6 header, set by the sender to a value that stays fixed for a flow (RFC 6437). RFC 6438 recommends that tunnel endpoints fill it in, so routers in the middle can hash on addresses plus flow label and still spread tunnelled traffic, without digging into the tunnel's payload. Linux exposes the choice as a sysctl, fib_multipath_hash_policy: 0 for layer 3 only (the default), 1 for layer 4, 2 for the inner layer 3 header of tunnelled packets, 3 for a custom field list.
Then there is the question of mapping the hash onto N paths. RFC 2992 analyses three methods, and the differences only show up when the set of paths changes:
Hardware usually takes a fourth route, called resilient or consistent hashing. The hash selects one of many buckets in a table, and each bucket names a next hop. When a next hop fails, only the buckets that named it are rewritten to point elsewhere; every other bucket, and every flow in it, stays put. The Linux kernel implements this as resilient next-hop groups. Its documentation draws a 20-bucket table where deleting next hop 3 rewrites exactly the four buckets that held it. FRRouting's zebra can make every multipath group it creates resilient, with a configurable number of buckets. You can test all four methods in the lab below.
The lab generates random flows between documentation addresses (clients in 192.0.2.0/24, servers in 198.51.100.0/24 and 203.0.113.0/24, ephemeral source ports, a few well-known destination ports) and hashes each one onto the next hops. Choose how many next hops, which header fields go into the hash, and how the hash is mapped to a next hop. Then fail a link and watch which flows move. The flows that were on the failed link must move; the interesting number is how many other flows moved for no reason.
| Flow (5-tuple) | Before | After |
|---|
ECMP has a trap that only appears when you stack it. Imagine a router that splits traffic across two next hops, A1 and A2, by computing hash mod 2. Every flow that lands on A1 has a hash whose remainder is 0. Now A1 also has two equal-cost uplinks and splits traffic the same way: same hash function, same fields, same seed, mod 2. Every flow it receives has remainder 0, so every flow goes to its first uplink. The second uplink carries nothing. Half the capacity at that tier is idle, and no amount of extra traffic will reach it.
This is called hash polarization, and it happens whenever successive routers make correlated choices. The usual fix is to make each router's hash different: mix a per-router seed into the hash (many platforms derive one from the router's ID or a MAC address), or use different fields or different bits of the hash result at different tiers. The instrument below sends the same flows through two tiers and lets you switch between one shared hash and a per-router seed.
ECMP assumes equal paths, but networks are not always so tidy. One spine may have lost a link; one uplink may be 100 Gb/s and its neighbor 40. Sending them equal shares overloads the weaker path. The fix is weighted ECMP: give each next hop a share of the hash space, or of the bucket table, in proportion to a weight. RFC 2992 notes that unequal regions in hash-threshold give exactly this, and Linux's ip route accepts a weight per next hop in a multipath route. FRRouting's static routes take a weight from 1 to 65535 per next hop, and its documentation notes the kernel itself only supports 8-bit weights, so zebra builds next-hop groups to carry the full range.
Weights come from configuration or from the control plane. FRRouting, for example, documents weighted ECMP using BGP link bandwidth: a router can weight its BGP next hops by a bandwidth value advertised with the routes, so traffic follows the capacity behind each path.
Most protocols only load-share over paths with exactly the best metric, because using a worse path risks a loop: if your neighbor's route to the destination runs back through you, sending it some traffic just bounces packets between you. EIGRP, documented in RFC 7868, can safely use a worse path because of its feasibility condition: a neighbor is a feasible successor only if its own reported distance is less than your best distance, which proves its path does not run through you. RFC 7868 notes that feasible successors can carry traffic "in the event equal or unequal cost load sharing was active." Cisco's implementation exposes this through a variance multiplier: paths whose metric is within that multiple of the best, and which meet the feasibility condition, are installed with traffic shares in inverse proportion to their metric.
It never became common outside EIGRP networks. Link-state protocols can compute loop-free alternates (chapter 07), but they use them as instant backups, not for load sharing. Where unequal sharing is wanted today, it is usually built from weighted ECMP plus traffic engineering (chapter 12).
Hashing balances the number of flows, not the number of bytes. If a few large flows (often called elephants) happen to hash onto the same link, that link can saturate while its neighbors idle, however well the hash spreads the flow count. RFC 7424 describes the problem for both ECMP and link aggregation and surveys fixes: detecting large flows and pinning them to lightly loaded links, rebalancing when the imbalance passes a threshold, and splitting traffic more finely. Each fix trades some of the simplicity of the plain hash for better balance.
Everything so far chose a route by destination alone. Sometimes that is the wrong question. A small office with two internet uplinks, one from provider A with addresses in 203.0.113.0/24 and one from provider B with addresses in 198.51.100.0/24, has two default routes. Destination routing can only pick one. But a reply to a connection that came in on provider B's address must go back out through provider B: provider A will, quite properly, drop packets whose source address it never assigned (anti-spoofing filters, see chapter 11). The router has to look at the source address to choose.
That is policy-based routing. Linux builds it from two pieces: several routing tables instead of one, and a routing policy database of rules that decides which table to consult. According to the ip-rule manual, the kernel starts with three rules: priority 0 looks up the local table (the router's own and broadcast addresses), priority 32766 looks up main (the table ip route shows by default), and priority 32767 looks up default, which is empty. Rules are tried from the lowest number upward. You add a rule such as "from 198.51.100.0/24 lookup table isp-b" at, say, priority 100, and give table isp-b its own default route via provider B. If a table has no matching route, the lookup moves on to the next rule.
Routers from every vendor have an equivalent, often called PBR, with match conditions on source, incoming interface, protocol or DSCP. It is powerful and easy to lose track of: a policy route bypasses the normal routing table, so it bypasses everything this chapter described. Keep policies few and documented.
Tables: local holds 192.0.2.1, 203.0.113.2 and 198.51.100.2 (the router itself). isp-b: 198.51.100.0/24 on wan-b, default via 198.51.100.1. main: 192.0.2.0/24 on lan, 203.0.113.0/24 on wan-a, 198.51.100.0/24 on wan-b, default via 203.0.113.1.