BGP policy: who pays whom

If BGP itself is new, start with dynamic routing on networks.jelia.nyc, which introduces BGP as the internet's policy protocol and lets you play with a best-path chooser. The previous chapter, BGP sessions, messages & iBGP, shows how two routers open a session and what the attributes look like on the wire. Here we assume both and ask a different question: given that BGP lets every network choose, what do networks actually choose, and why?

The internet is made of tens of thousands of autonomous systems (ASes), each a network under one administration with its own routing policy. Almost every link between two of them falls into one of a few business shapes. The most common is transit. A customer AS pays a provider AS, and in return the provider carries the customer's traffic to and from the entire internet. A university buys transit from a regional network; the regional network buys transit from a larger one. Money flows up the hierarchy, and reachability flows down.

The second is settlement-free peering. Two networks of roughly comparable size notice that a lot of traffic flows between their customers. Rather than both paying a provider to carry it, they connect directly and exchange that traffic for free. The key word is between their customers: a peer gives you access to itself and its customers, never to the rest of the internet. If it did, you would be getting free transit, and transit is exactly what it sells.

The third is paid peering, where one side pays the other but the routes exchanged are the same as in ordinary peering: only the seller's own and customer routes. It shows up when traffic between two networks is heavily one-sided. From BGP's point of view it behaves like peering, so this chapter treats it as one.

Nothing in the BGP protocol records any of this. There is no "customer" bit in an UPDATE message. The relationship lives in a contract and in each operator's router configuration, and the internet's overall shape is the sum of tens of thousands of private decisions. Researchers recover the relationships by studying which paths appear in public BGP data: CAIDA publishes an inferred customer, provider and peer label for links between ASes, which is how most studies of internet structure begin. Chapter 10 takes up the large-scale shape: tier-1 networks, exchange points and the growth of the table.

Once you know who pays whom, the routing policy almost writes itself. Every route an AS learns came from a customer, a peer or a provider. The question for each route is: if I pass it on, who will send traffic through me because of it, and who pays for that traffic?

• Routes learned from a customer go to everyone: providers, peers and other customers. Any traffic that follows them flows to a customer, who pays for it.
• Routes learned from a peer go only to customers. Announcing them to a provider or another peer would invite outsiders to send traffic across your network to that peer, for which nobody pays you.
• Routes learned from a provider go only to customers. Announcing them upward or sideways would mean paying your provider to carry other people's traffic.

And when an AS has several routes to the same destination, it ranks them by money too: a customer route first (traffic sent there earns revenue), then a peer route (free), then a provider route (costs money). Operators implement that ranking with LOCAL_PREF, a number attached to each route as it arrives. Higher wins, and it is the very first thing BGP compares.

Follow those rules everywhere and every path on the internet takes the same shape. Read a path from the source network to the destination: it climbs through zero or more customer-to-provider links, crosses at most one peering link, then descends through zero or more provider-to-customer links. It looks like a mountain, possibly with a flat top. It never dips down and climbs back up, because the AS at the bottom of that dip would be carrying traffic between two networks that do not pay it. A path with that dip is called a valley, and a path without one is valley-free.

In 2000 Lixin Gao and Jennifer Rexford proved why this matters beyond money. BGP has no global coordinator: every AS applies its own private preferences, and in general that can make the protocol oscillate forever, each change triggering another. Gao and Rexford showed that if every AS prefers customer routes over peer and provider routes, follows the export rules, and no AS is indirectly its own provider (the customer-provider links contain no cycle), then BGP is guaranteed to settle into a stable set of routes, whatever order messages arrive in and after any failure. The internet's commercial instincts happen to be exactly what keeps its routing protocol from thrashing.

Try it in the instrument: originate at AS64501 and make AS64502 the leaking AS. AS64502 is a small network with two providers. It learns AS64501's route from one provider, AS64498, and, ignoring the rules, re-announces it to its other provider, AS64499. To AS64499 that announcement arrives from a customer, so it outranks the peer route it already had, even though the path is longer. AS64499 then does what it should do with a customer route and tells everyone. Within a round or two, much of the graph is sending traffic for AS64501 down into the small network and back up again.

That is a route leak, and RFC 7908 defines it as the propagation of an announcement beyond its intended scope, in violation of the policies of the networks involved. The version you just built is what the RFC calls a "hairpin turn": a multihomed customer passing one provider's route to another. The damage is rarely about reachability. Packets still arrive, mostly. The damage is that a network sized for its own traffic suddenly carries a slice of somebody else's, and links saturate, latency jumps, and packets are dropped along the detour.

Leaks are almost always configuration mistakes: a missing export filter, a policy applied to the wrong session, a router restored with a default configuration that exports everything. The rules in this chapter are the definition of what is intended, and the defences, from filtering customers' announcements to the newer BGP Roles of RFC 9234 and ASPA, are built to enforce them automatically. They belong to Chapter 11.

Traffic engineering means moving traffic between your links on purpose: to use the cheaper one, to keep a congested one under its limit, or to keep a backup idle. There are two directions, and they are not equally easy.

Outbound traffic, the packets your network sends, is entirely under your control, because you choose the route. The main tool is LOCAL_PREF. Raise it on routes from the provider you want to use and every router in your AS prefers that exit for those destinations. LOCAL_PREF is shared inside your AS over iBGP and, by RFC 4271's rule, never sent to an external neighbor: it is your private opinion. Tie-breaking does the rest. When LOCAL_PREF and AS-path length are equal, each router picks the exit with the lowest internal (IGP) cost to the BGP next hop.

That last step has a nickname: hot-potato routing. A large network with exits in many cities hands traffic to the next network at the exit nearest to where it entered, to get it off its own backbone as quickly as possible. It is efficient for the network doing it and has a striking side effect: the two directions of a conversation often take different paths. Your request may cross an ocean on one network's cables and the reply on the other's. Traceroute shows you only the outbound half, which is one reason Chapter 15 calls it a liar about the way back.

This is the order in RFC 4271, section 9.1. Real routers add a few vendor-specific steps (some implementations put a local, per-router weight before LOCAL_PREF, or prefer the oldest of two external routes to reduce churn), but the spine is the same everywhere. Chapter 3 covers how BGP's winner then competes with other protocols for a place in the forwarding table.

Inbound traffic is the hard direction. Packets coming to you follow routes chosen by other networks, using their LOCAL_PREF, their tie-breaks and their policies. You cannot set anything inside their routers. All you can do is change what you announce, so that their decision process lands where you want. There are five common tools, and each has a characteristic limit.

AS-path prepending. When you announce your prefix to one provider, repeat your own AS number several times: 64511 64511 64511 instead of 64511. The path through that provider now looks longer to everyone, and networks choosing by AS-path length move to the other one. It is crude but effective, and it has a hard limit: path length is step 2. Any network whose LOCAL_PREF prefers the prepended route, such as that provider itself (you are its customer) or its other customers, ignores the padding completely. Once every network that ties on LOCAL_PREF has moved, further prepends move nothing more, and each one makes your route look stranger in every table on earth.

MED (MULTI_EXIT_DISC). If you have two links to the same provider, say in two cities, you can attach a lower MED to the announcement on the link you prefer. The provider compares MEDs at step 4 and sends traffic to the lower one, overriding its own hot-potato preference. MED has two limits. By default it is only compared between routes from the same neighbor AS, so it does nothing to choose between two different providers. And it is non-transitive: RFC 4271 says a MED received from one neighbor must not be passed on to other ASes, so it influences only your direct neighbor. Some providers ignore it, or honor it only by agreement.

More-specific prefixes. Announce your aggregate to both providers, and two halves of it to only one. Longest-prefix match beats every BGP preference, because the halves are different destinations as far as the forwarding table is concerned. Every router that has the more-specific uses it, so traffic swings almost completely to the provider announcing the halves. That power comes at a price paid by everyone: each extra prefix occupies memory in every full-table router in the world. Networks commonly refuse anything longer than a /24 in IPv4 or a /48 in IPv6 (RFC 7454 notes this practice), so a /24 cannot be split this way at all. More-specifics also never fail over gracefully: if the link carrying them breaks, traffic falls back to the aggregate, which is the point, but if they leak further than intended they can drag traffic somewhere unexpected.

Selective announcement. Simply do not announce a prefix to one provider. Traffic for it can then only arrive the other way. It is clean and absolute, and the cost is that the prefix loses its redundancy.

Communities. Many providers publish a menu of BGP communities their customers can attach to a route: "set my LOCAL_PREF inside your network to the backup level," "prepend twice when you announce me to your peers in Europe," "do not announce me to this peer." Because the community changes the provider's own policy, it reaches the one lever prepending cannot: LOCAL_PREF. The next section is about communities themselves.

Start the playground from its defaults and read the source table. About 40 percent of traffic arrives on L1, 15 percent on L2 and 45 percent on L3. Here is why, one source at a time.

• Transit 2 is a provider of B but not of A. It hears your route from B as a customer route (LOCAL_PREF 200) and from Transit 1 as a peer route (100). LOCAL_PREF decides: B.
• Transit 1 is a provider of both A and B, so it has two customer routes, both two ASes long. MED cannot help, because they come from different neighbor ASes. Its IGP costs to the two exits are equal, so the last step, lowest router ID, picks A. Then inside A, the traffic enters in city 1 and leaves by the nearest link, L1: hot potato.
• Content peers with both providers. Equal LOCAL_PREF, equal length; its own IGP cost to B is lower, so B.
• Access 1 and Access 2 are single-homed; they have one route each. Access 3 is multihomed and ties all the way to router ID, which picks A.

Now suppose L1 is filling up. Set Prepend toward A to 1. Transit 1 and Access 3 now see a shorter path via B and move: L1 drops to zero and L3 jumps to 85 percent. Prepending is a coarse tool: it moves whole networks at once. Put it back to 0 and instead set Prepend toward B to 1. This time Content moves to A, and you land near 40/30/30. Now push Prepend toward B to 5 and notice that Transit 2 and Access 2 do not move at all: their choice was made at LOCAL_PREF, one step before path length is ever looked at.

Finally, return to defaults and set the MED on L2 to 50. Everything Provider A carries now uses L1, because MED is compared before IGP cost and overrides A's hot-potato instinct. Choose Send it to B under backup community and L3 falls silent: B now prefers to reach you through its peer A, which only B's customers will follow, and Transit 2 reaches you through Transit 1. Then cut both links to A and watch L3 carry everything again. That is what a backup link is supposed to do.

A community is a label attached to a route, defined in RFC 1997 in 1996. It is a 32-bit value written as two 16-bit halves, ASN:value, where by convention the first half is the AS number of the network that defines the meaning. The protocol attaches no meaning to most values. Their power is that they travel with the route and any router's policy can match on them, so they become a shared vocabulary between networks.

Operators use them in two directions. Informational communities are added by a network to describe a route: "learned from a customer," "learned in Frankfurt," "learned from peer number 12." Downstream, they make filtering and debugging easier, and they are how a network implements its own export rules internally. Action communities are sent by a customer to ask the provider to do something: lower my LOCAL_PREF to the backup level (the approach described in RFC 1998, which is where the playground's backup option comes from), prepend when announcing me to a given peer, or do not announce me there at all.

A handful of values are well-known, reserved in the 65535:x range and recognized by every implementation that supports them.

The classic 32-bit community has a problem: the first half is only 16 bits, and many AS numbers are now 32 bits long. An operator with AS65551 cannot write its own number into a classic community. Large communities, RFC 8092 (2017), fix that with a 12-byte value: a 4-byte Global Administrator (the defining AS) and two 4-byte local data parts, written 65551:1:200. A provider might publish "65551:1:n means set LOCAL_PREF to n," for instance. RFC 8195 collects usage examples.

BLACKHOLE deserves its own paragraph because it is a defensive tool with a sharp edge. When one address in your network is being flooded by a distributed denial-of-service attack, the flood can saturate your links and take down everything behind them. With a provider that has agreed to honor it, you announce a very specific route covering just the victim address, typically a /32 in IPv4 or a /128 in IPv6, tagged 65535:666. The provider then discards traffic to that address at its own edge, before it ever reaches your link. The victim address goes dark, which completes the attacker's goal for that one address, but everything else in your network stays up. RFC 7999 says the receiver should accept such routes only for space the customer is authorized to announce, only by prior agreement, and should add NO_EXPORT or NO_ADVERTISE so the blackhole stays local. Accepting a blackhole request for someone else's address would let one customer black out another's, which is why those checks matter.

In a router's configuration, everything in this chapter becomes a small number of route maps or filters applied per session. On each customer session, an import policy accepts only the prefixes that customer is entitled to announce, sets LOCAL_PREF to the customer level, tags the route with an informational "from a customer" community and acts on any action communities the customer sent. On each peer session, the import policy sets the peer level and tags "from a peer"; on each provider session, the provider level and "from a provider." The export policy on peer and provider sessions is then one line of logic: announce only routes tagged "from a customer," plus your own.

That design is worth noticing. Because the export rules are written in terms of tags attached on the way in, a mistake on one session cannot leak routes on another, and a new session is safe as soon as it is given its role. The common leak scenario, a session with no export policy at all, is exactly what newer protections address: RFC 8212 changed the default so that a session with no policy sends and accepts nothing, and BGP Roles (RFC 9234) let two routers agree on the relationship and mark routes so a leak can be detected one hop later. Both are covered in Chapter 11.