BGP sessions, messages & iBGP

If BGP's job as a policy protocol, the one that chooses between paths by business rules rather than by shortest distance, is new to you, start with dynamic routing on networks.jelia.nyc, which introduces it and lets you play with a best-path chooser. This chapter assumes that and looks at the machinery underneath.

The interior protocols in Part II discover their neighbors on their own. An OSPF router multicasts hellos onto every link and makes friends with whoever answers (Chapter 05). BGP does nothing of the kind. Every BGP neighbor, called a peer, is configured by hand: this address, this AS number, these filters. Two networks that connect have usually signed something first, and the configuration is the technical half of that agreement.

Once configured, two peers open an ordinary TCP connection to port 179 and keep it open for as long as the relationship lasts, often for months. That choice, made in the first BGP specification in 1989, shapes everything else about the protocol.

• Reliability is somebody else's problem. TCP delivers bytes in order, retransmits what is lost and slows down when the receiver is busy. BGP has no sequence numbers, no acknowledgments and no retransmission logic of its own.
• Updates are incremental. Because nothing is lost, a router tells its peer the whole table once, when the session comes up, and after that only sends changes: this prefix is new, that one is gone. RIP re-sends its whole table every 30 seconds and OSPF refreshes every LSA every 30 minutes; BGP never refreshes anything on a timer. With about a million routes in the global table, that is the only approach that scales.
• One session, one state. Everything a router has learned from a peer is tied to that TCP connection. If the session ends, every route learned over it is withdrawn at once. That makes a session reset the most expensive event in BGP, which is why so much of the protocol's recent history is about avoiding them.

The price of TCP is that a BGP speaker cannot use it to notice a dead peer quickly. TCP can sit for many minutes on a connection whose other end has vanished. So BGP adds a heartbeat of its own, the KEEPALIVE message, and a hold timer that tears the session down if nothing at all arrives for too long.

RFC 4271 §8 describes a BGP session as a finite-state machine: a small set of named states, and for each state a list of events that move it to another. A router runs one machine per configured peer. Six states matter.

The confusing name is Active. It sounds like the good state, and it is nearly the opposite: a router in Active has tried to reach its peer, failed, and is waiting to try again. A session that sits in Active, or flips between Connect and Active forever, is a TCP problem, not a BGP one. The usual causes are easy to list and tedious to find.

• The two ends disagree about addresses: R1 is configured to peer with R2's loopback, but R2 opens connections from its interface address, and R1 rejects a connection from an address it was not told about.
• Something between them drops TCP port 179: an access list, a firewall, a security group.
• There is no route to the peer's address. Peering between loopbacks needs the interior protocol to carry those loopbacks first.
• The peer is two or more hops away but the session is configured as directly connected. Many implementations expect eBGP peers to be on a shared link unless told otherwise (FRRouting's ebgp-multihop setting is one example).
• The other side simply has no configuration for this peer yet.

Problems inside BGP look different. If TCP works but the OPEN is wrong, the session gets as far as OpenSent, a NOTIFICATION goes out, and the machine drops back to Idle, then retries a little later and fails the same way. Operators call that a flapping session. The most common OPEN mistake is a wrong AS number: R2 was told to expect R1 in AS 64496 and R1's OPEN says something else, so R2 answers with NOTIFICATION code 2, subcode 2, "Bad Peer AS."

A mismatched hold time, by contrast, is not an error. Each side proposes one in its OPEN, and both use the smaller.

RFC 4271 suggests a hold time of 90 seconds with keepalives every 30. Many implementations default to 180 and 60 instead; the sample show bgp neighbors output in FRRouting's documentation, for example, reports "Hold time is 180, keepalive interval is 60 seconds." Three minutes is a long time to keep sending traffic toward a router that has died, which is why fast detection is usually handed to BFD (Chapter 07) rather than squeezed out of BGP's own timers.

Everything BGP says is one of five message types, and every message starts with the same 19-byte header. Because TCP delivers a stream of bytes rather than separate packets, the header's main job is to tell the receiver where each message ends.

The marker is sixteen bytes of 0xFF. It once had room for authentication data; RFC 4271 fixed it at all ones, and today it serves as a sanity check that the receiver is still in step with the stream. The length covers the whole message including the header, and must be between 19 and 4,096. The type says which of the five messages follows. The bytes above are a complete KEEPALIVE: a header with nothing after it.

OPEN is where the two routers negotiate. Beyond the fixed fields it carries capabilities (RFC 5492): short codes that announce optional features, each used only if both sides advertise it. Capabilities are how BGP has grown for three decades without a version 5. Multiprotocol support, route refresh, four-byte AS numbers, ADD-PATH and larger messages are all switched on this way.

The My AS field is only two bytes wide, because when BGP-4 was written nobody expected more than 65,535 networks. When four-byte AS numbers arrived, RFC 6793 kept the field and added a capability, code 65, carrying the real four-byte number. A router whose number does not fit puts 23456, AS_TRANS, in the old field. Old routers see a valid-looking two-byte AS; new routers read the capability. AS_TRANS appears in AS paths for the same reason: an old router passes on a path in which every unfittable AS has been replaced by 23456, and a separate attribute, AS4_PATH, carries the true numbers past it.

NOTIFICATION is BGP's only way of reporting an error, and it is final: the sender closes the TCP connection right after sending it, and both sides throw away every route learned over the session. The code says which part of the protocol went wrong.

An UPDATE has three parts. First a list of withdrawn routes: prefixes the sender no longer reaches. Then the path attributes: facts about a path, such as the ASes it crosses and where to send the packets. Last the NLRI, network layer reachability information, which is BGP's name for the list of prefixes that share those attributes. One UPDATE can announce hundreds of prefixes at once if they all came by the same path, and that packing is a large part of how BGP stays efficient.

Prefixes are packed tightly: one byte for the prefix length, then only as many address bytes as the length needs. 203.0.113.0/24 takes four bytes on the wire (18 CB 00 71: length 24, then 203.0.113), and 198.51.100.128/25 takes five.

Each attribute starts with a flags byte, a type code and a length. Two flag bits sort attributes into four kinds, and those kinds decide what a router does with an attribute it has never heard of.

The optional-transitive rule is what made BGP so extensible: a new attribute can cross the internet through routers that are years older than it, as long as the routers at each end understand it. The fourth flag bit, 0x10, means the length field is two bytes instead of one.

• ORIGIN says how the route entered BGP: 0 IGP (configured as a network statement), 1 EGP (from the long-dead protocol BGP replaced) or 2 INCOMPLETE (redistributed from somewhere else). It is a late tiebreaker in path selection.
• AS_PATH lists the ASes the route has crossed, newest first. Each eBGP speaker prepends its own AS on the way out. It serves as both a distance (shorter is preferred) and a loop detector: a router discards any route whose path already contains its own AS. It is built from segments, and the AS_SEQUENCE segment (type 2) is the one in use. The AS_SET segment (type 1), an unordered bag used when aggregating routes, was deprecated by RFC 9774, which also tells receivers to treat a route carrying one as withdrawn.
• NEXT_HOP is the address packets should be sent to. On eBGP it is normally the sender's own address on the shared link.
• MED (multi-exit discriminator) is a hint to a neighboring AS about which of several links into you to prefer. LOCAL_PREF is a preference shared only inside one AS, and the first thing path selection compares. Both are policy tools, and Chapter 09 is where they are used, along with communities, the 32-bit tags networks attach to routes to request treatment from each other.

BGP-4 was written for IPv4: the NLRI field at the end of an UPDATE can only hold IPv4 prefixes, and NEXT_HOP is exactly four bytes. RFC 4760 removed that limit without changing the message format. It added two optional non-transitive attributes. MP_REACH_NLRI (type 14) carries an address family, a next hop of any length and a list of reachable prefixes; MP_UNREACH_NLRI (type 15) carries withdrawals. The family is named by two numbers: the AFI (address family identifier) says what kind of address, and the SAFI (subsequent address family identifier) says what the routes are for.

Each family is switched on separately with capability 1 in the OPEN, so one TCP session can carry IPv4, IPv6 and VPN routes side by side, each with its own table. That is why BGP has become the general-purpose route distribution protocol of large networks: anything that can be written as a prefix with attributes can ride on it. RFC 8950 adds one more combination worth knowing: IPv4 prefixes announced with an IPv6 next hop, so a link or a whole fabric can carry IPv4 traffic without any IPv4 addresses on its interfaces.

An AS with several border routers has a problem. R1 hears a route from a neighbor network; R5, on the other side of the country, needs to know about it too. The interior protocol (OSPF or IS-IS) cannot carry it: those protocols are built for a few thousand internal prefixes, not a million external ones with policy attached. So the routers inside the AS run BGP among themselves. That is iBGP: the same protocol, between routers with the same AS number.

iBGP behaves differently in three ways. A router does not prepend its AS on iBGP, so the AS_PATH is unchanged inside the network. It does not change NEXT_HOP either: R5 learns a route whose next hop is R1's external neighbor, an address R5 can only reach if the interior protocol knows how to get there. Operators fix that by configuring next-hop-self on border routers, which writes the border router's own loopback into NEXT_HOP before passing the route inward. And, the rule that matters most:

The consequence is that every router that learns an external route must tell every other router in the AS itself, directly. Every pair of BGP routers in the AS needs its own session: a full mesh.

There are two ways out, and both bend the rule in a controlled way.

Route reflectors (RFC 4456) are the common one. A few routers are designated reflectors, and the rest are their clients. A reflector is allowed to pass iBGP routes on: a route from a client goes to all other clients and to the reflector's non-client peers; a route from a non-client goes to clients only; a route from eBGP goes to everyone. Clients need nothing special; they just peer with the reflectors instead of with each other. Reflectors are normally deployed in pairs or more, so one failure doesn't cut the clients off.

Bending the rule brings loops back, so reflection adds two optional non-transitive attributes to prevent them. ORIGINATOR_ID records the BGP Identifier of the router that first injected the route into the AS; a router that sees its own identifier there ignores the route. CLUSTER_LIST records every reflection cluster the route has passed through; a reflector that finds its own cluster ID in the list drops the route.

Confederations (RFC 5065) take the other approach. The AS is split into several member ASes, usually numbered from the private range 64512 to 65534, each a full mesh internally, joined by sessions that behave like eBGP. Inside the confederation the AS_PATH records member ASes in special segments, AS_CONFED_SEQUENCE, which are stripped at the edge, so the outside world sees one AS. LOCAL_PREF and NEXT_HOP pass between members unchanged, as if it were all iBGP. Confederations are rarer than reflectors, but they turn up after mergers, when two networks become one AS and their internal structure survives.

The original rule for a malformed UPDATE was simple: send NOTIFICATION code 3 and close the session. That protects the receiver from bad data but punishes everything else on the session. One wrongly encoded attribute on one route, perhaps set by a router several networks away and passed along by an optional-transitive rule, would tear down a session carrying a million good routes. When the session came back, the same UPDATE arrived again, and the session dropped again. Real incidents of this kind are why RFC 7606, "Revised Error Handling for BGP UPDATE Messages," was written in 2015.

RFC 7606 sorts errors into three responses, from mildest to most severe.

• Attribute discard. The bad attribute cannot affect route selection, so ignore it and keep the route. A LOCAL_PREF received from an external peer, where it should never appear, is one example.
• Treat-as-withdraw. The attribute matters, so the routes in this UPDATE cannot be trusted. Handle them as if the UPDATE had withdrawn them, and keep the session. A bad ORIGIN value, a NEXT_HOP of the wrong length, a missing mandatory attribute and a malformed AS_PATH all land here.
• Session reset. Only when the receiver cannot even tell where the message's parts begin and end, for example when the withdrawn-routes and attribute lengths add up to more than the message. Then nothing in it can be located reliably, and closing the session is the only safe choice.

Treat-as-withdraw only works if the receiver can still find the list of prefixes. So RFC 7606 also asks senders to put MP_REACH_NLRI or MP_UNREACH_NLRI first among the attributes, and to put only one kind of route list in each UPDATE. The decoder above applies these rules: try the "bad ORIGIN" sample and then the "lengths lie" one.

The same instinct, avoid resets, runs through several other extensions. ROUTE-REFRESH lets a router re-read a peer's routes without restarting the session. Graceful restart (RFC 4724) lets a router's control plane restart while its neighbors keep forwarding along the old routes for a while. And graceful shutdown (RFC 8326) lets an operator warn neighbors before taking a session down for maintenance: routes are tagged with the well-known community GRACEFUL_SHUTDOWN, 65535:0, and receivers lower their preference, so traffic has already moved to other paths when the session finally closes.