Chapter 10 · Part III · BGP & the internet

The shape of the internet: transit, peering & IXPs

The internet is not one network with a center. It is tens of thousands of separately owned networks, each with its own number, joined by contracts: some pay others to carry their traffic, some swap traffic for free, and many meet on shared switches in a few hundred buildings. This chapter draws that graph, shows why its shape keeps changing, and counts the routes it costs every router to carry.

1,050,000IPv4 routes in the global BGP table in January 2026, with about 242,000 more for IPv6 (Geoff Huston, potaroo.net)
232possible AS numbers since BGP moved to four-byte ASNs (RFC 6793); the original two-byte field allowed only 65,536
0AS numbers an exchange's route server adds to the AS path: RFC 7947 says it SHOULD NOT prepend its own

Common mix-up: "tier 1" does not mean "biggest." It means transit-free: a network that reaches every destination through its own customers and through peers, without paying anyone for transit. A content network can carry more traffic than many tier 1s and still buy some transit, and a tier 1 can be modest in size. The label describes a position in a web of contracts, not a size.

A network of networks

About 78,000 autonomous systems

If BGP, peering and anycast are new to you, start with dynamic routing on networks.jelia.nyc, which introduces BGP as the internet's policy protocol and shows anycast in one picture. This chapter assumes that and zooms out to the whole graph.

The unit of the internet's map is the autonomous system, or AS. RFC 1930 defines it as "a connected group of one or more IP prefixes run by one or more network operators which has a SINGLE and CLEARLY DEFINED routing policy." In practice an AS is an organization's network as the rest of the internet sees it: an access provider, a university, a cloud, a bank, a content network. Each has an AS number (ASN), and each speaks BGP to its neighbors, announcing the address blocks it originates and passing on the ones it is willing to carry. Chapter 8 covers how those sessions work.

How many are there? Geoff Huston's annual report, measured from one vantage point, counted about 77,900 ASes visible in the IPv4 BGP table in January 2026. Only about 11,400 of them were transit ASes, meaning they appear in the middle of some path and so carry traffic for others. The other 66,500 were stubs: networks that appear only at the end of paths, originating their own addresses and nothing else. In IPv6 the same report counted about 36,100 ASes. Most of the internet, by count, is edges.

Between any two ASes there is usually one of three business relationships, and Chapter 9 is all about them: a customer pays a provider for transit to the whole internet; two peers exchange traffic between their own customers, usually without payment. Those contracts, not distance, decide which paths exist. This chapter is about the shape they give the graph.

Go deeper: from two-byte to four-byte AS numbers

BGP-4 (RFC 4271) carried AS numbers in two bytes, so there could only ever be 65,536 of them, and some were reserved. By the mid-2000s the registries could see the end of that pool. RFC 4893 (2007), later replaced by RFC 6793 (2012), widened the field to four bytes: 232, about 4.29 billion values.

The hard part was the transition, because the internet cannot upgrade every router at once. The solution has three pieces. Two speakers that both support four-byte ASNs say so when the session opens and then simply send four-byte paths. Toward an old speaker, a new one writes its AS path in two-byte form and replaces any number that does not fit with a reserved placeholder, AS_TRANS, which is AS 23456. The true four-byte path rides along in a new optional transitive attribute, AS4_PATH, which old routers pass on without understanding. When the route reaches a new speaker again, it rebuilds the real path from the two.

Old two-byte numbers stay valid: they are four-byte numbers whose top two bytes are zero. That is why you will see some networks described in "asdot" form, such as 1.0 for 65,536, and most tools today print the plain decimal.

AS 65536 → AS_TRANS 23456 + AS4_PATH
A worked example with a documentation ASN (RFC 5398 reserves 65536–65551 for examples). 65,536 needs more than 16 bits, so the old AS_PATH field cannot hold it; AS_TRANS stands in, and the four-byte value is restored downstream.
The pyramid, and why it is blurry

Tiers are a loose model

The traditional picture is a pyramid. At the top is a small group of tier 1 networks. Each one buys transit from nobody: it can reach every address on the internet through its own customers or through settlement-free peering with every other tier 1. Below them, tier 2 networks are regional or national providers that peer where they can but still buy transit from someone above to reach the rest. At the bottom, tier 3 and stub networks buy all their connectivity.

The model is useful because it explains a basic shape: a packet between two distant stub networks usually climbs from customer to provider, crosses at most one peering link near the top, and descends from provider to customer. That is the valley-free pattern, and Instrument 1 below computes it.

But the tier labels are fuzzy for several reasons. Contracts are private, so nobody outside knows for sure who pays whom; researchers infer relationships from public BGP data, as CAIDA does in its AS relationships dataset, and inferences can be wrong. Relationships are not all-or-nothing: a network can buy partial transit (routes to one region only), pay for peering, or peer in one country and sell transit in another. And "transit-free" depends on the day: if two would-be tier 1s stop peering in a dispute, each loses reachability to the other's customers until someone pays. For these reasons this chapter will not name or rank commercial networks as tier 1; the CAIDA dataset and its "customer cone" ranking are the place to look for evidence-based numbers.

The internet has flattened. Through the 2000s, large content and cloud networks began building their own long-haul backbones and connecting directly to the access networks where their users are, at exchange points and through private links. A 2008 measurement study by Gill, Arlitt, Li and Mahanti called this "the flattening internet topology," and a 2010 study by Labovitz and colleagues found, in traffic data from many providers, that a growing share of inter-domain traffic flowed directly between content or hosting networks and consumer networks, bypassing the traditional top tier. Huston's 2025 report describes the same economics from the routing side: content and services are increasingly delivered by "fewer, but far larger, service platform operators." The pyramid still exists for reachability, but most of the bytes people actually use travel a much shorter, flatter path.

Go deeper: the customer cone, and how relationships are inferred

An AS's customer cone is the set of ASes it can reach by going only downhill: its customers, their customers, and so on. It is a good proxy for a network's position in the transit market, and CAIDA publishes rankings by cone size. Inferring the cone requires knowing which links are customer-to-provider, which is not announced anywhere.

The classic method comes from Lixin Gao's 2001 paper on inferring AS relationships: in a valley-free path, the highest-degree AS is probably the top of the hill, links before it go uphill and links after it go downhill. Later work, notably Luckie and colleagues at CAIDA in 2013, improved accuracy by first finding a clique of transit-free networks at the top and then propagating constraints down, and validated the results against operator-reported data and BGP communities. The answer is still an inference with error bars, which is one more reason tiers are a model, not a fact.

Instrument 1

Hierarchy explorer

A synthetic internet built from documentation AS numbers: three transit-free networks at the top that all peer with each other, four regional providers, one content network and six stubs. Solid lines are customer-to-provider links (the provider is the upper end); dashed lines are peering. Tap a network to choose where a packet starts, then tap another for where it goes, or use the menus. The instrument runs BGP route selection with the standard commercial rules (prefer routes learned from customers, then peers, then providers; then the shorter AS path; export peer and provider routes only to customers) and draws the path that results in cobalt. If the plain shortest path through the graph is different, it is drawn dashed in red, with each step marked ↑ (to a provider), → (to a peer) or ↓ (to a customer). Then switch on the exchange point and watch paths shorten.

Path BGP chooses–
Its steps–
Shortest graph path–
Its steps–

Exchange members when switched on: AS64500, AS64503, AS64510, AS65538 and AS65541, all peering through a route server, AS64511. Notice that AS64511 never appears in a path.

One switch, hundreds of networks

Internet exchange points

Peering two networks needs a physical link between them. Doing that one cable at a time scales badly: a network that wants to peer with 200 others would need 200 links. An internet exchange point (IXP) solves the physical half. It is a shared layer-2 switching fabric, usually Ethernet, spread across one or more data centers in a city. Each member brings a router, buys a port on the fabric and gets an address on the exchange's shared subnet. From then on every other member is one hop away on the same LAN.

The fabric does not decide anything about routing. Members still choose who they exchange routes with, and there are two ways to do it.

  • Bilateral peering. Two members agree to peer and configure a BGP session directly between their routers across the fabric. Each pair negotiates its own terms. With N members who all want to peer with each other, that is N(N−1)/2 sessions.
  • Multilateral peering through a route server. The exchange runs a route server, a BGP speaker that is not a router at all. Each member opens one session to it (usually two, to a redundant pair), sends its routes, and receives the routes of every other member that agreed to share with it. RFC 7948 compares it to an eBGP version of the iBGP route reflector.

Two details make the route server work, and both are in RFC 7947. First, the route server is not in the forwarding path: it passes each route on with the NEXT_HOP unchanged, so packets go straight from one member's router to the other's across the fabric. Second, it does not prepend its own AS number to the AS path, because it is not a network the traffic crosses, and adding a hop would make paths through the exchange look longer than they are. To a member's router, a route learned from the route server looks exactly as if it came straight from the member that originated it.

IXP switching fabric shared LAN 192.0.2.0/24 route server AS64511 AS64500 AS64503 AS64510 AS65541 packets: member to member dashed: one BGP session each to the route server
Control plane and data plane come apart at an exchange. Routes flow through the route server (dashed, teal); packets never do. They cross the fabric directly from one member's router to another's (solid, cobalt), because the route server leaves NEXT_HOP pointing at the originating member.
300 × 299 ÷ 2 = 44,850 sessions vs 2 × 300 = 600
A worked example of why route servers exist: 300 members in a full bilateral mesh, against one session each to two route servers. With bilateral peering the number of sessions grows with the square of the membership; with route servers it grows linearly. Most members do both: route-server peering for the long tail, bilateral sessions with the few networks they exchange the most traffic with.

Where do members find each other? PeeringDB is the public registry: in its own words "a freely available, user-maintained, database of networks," exchanges and facilities. A network's PeeringDB record lists which exchanges and buildings it is present in, its ASN, its peering policy (commonly described as open, selective or restrictive) and contacts. Operators check it before ordering a port, and many automate their route-server and bilateral configurations from it.

The alternative to the shared fabric is a private network interconnect (PNI): a dedicated cross-connect cable between two networks' routers in the same building. A PNI makes sense once two networks exchange enough traffic to fill whole ports, because then the shared fabric's port fees buy nothing extra, and a dedicated link isolates the pair from congestion or outages on the exchange. Big content networks and big access networks are often joined by many parallel PNIs.

How you reach another networkWho paysRoutes you getScales to
TransitYou pay your provider, usually per Mb/s at the 95th percentileThe whole internetEverything, at a per-unit price
IXP, route serverYou pay the exchange for a port; peers usually settlement-freeThe routes of every member that shares with the route serverHundreds of peers per port
IXP, bilateralSame port; one session per agreementThat peer's own and customer routesAs many as you negotiate
Private interconnect (PNI)Cross-connect and router ports; split by agreementThat peer's own and customer routesOne peer, but whole ports of capacity
Go deeper: path hiding, per-client RIBs and filtering at the route server

A route server runs the BGP decision process on behalf of its clients, and that creates a subtle problem RFC 7947 calls path hiding. Suppose two members both offer a route to the same prefix, and the route server picks one as best. If the member that originated the best route has told the route server not to share it with a third member, the third member should still get the second-best route. A simple route server with a single shared best-path table would send nothing, and the third member would lose a path it was entitled to. RFC 7948 notes that the only fix that needs nothing from the clients is for the route server to keep a separate table (Loc-RIB) per client, at a cost in memory and CPU that grows with the number of paths times the number of clients.

Members steer what the route server does with their routes using BGP communities (RFC 1997), for example "do not announce to this member." Because the route server sees every member's routes, it is also the natural place to filter: RFC 7948 describes building filters from Internet Routing Registry data so a member cannot announce space it has no business announcing. Chapter 11 covers origin validation with RPKI, which exchanges increasingly apply too.

The exchange LAN itself needs care. RFC 7454 recommends that networks not accept more-specific routes for the IXP LAN prefix from outside, since someone announcing them could pull exchange traffic away from the fabric.

Following the money

The cost logic of peering

Why does a network peer at all? Mostly because transit is billed by volume and an exchange port is billed by capacity. Transit is commonly priced per megabit per second per month, measured at the 95th percentile of five-minute samples, so the bill rises in a straight line with traffic. A peering port costs roughly the same each month whatever you send through it, up to the port's capacity. Below some volume, transit is cheaper; above it, peering wins. That crossover is the break-even point.

T* = (port + cross-connect) ÷ price per Mb/s
The volume at which a month of peering costs the same as a month of transit for the same traffic. With illustrative numbers, a $2,000 port and a $300 cross-connect against transit at $0.40 per Mb/s gives T* = 2,300 ÷ 0.40 = 5,750 Mb/s, about 5.8 Gb/s. Above that, every extra megabit moved by peering is free until the port fills.

Real decisions are messier than one formula. Peering also needs a router at the exchange and backhaul to it; a port is shared by traffic to many peers, so its cost is spread across all of them; the traffic you can move depends on which networks show up and agree to peer; and there are reasons to peer that are not about price at all, such as lower latency, more control over the path and resilience when a transit provider has a bad day. Equally, you never stop buying transit, because peering only reaches the peers and their customers. The model below is deliberately small so you can see the shape of the trade.

Instrument 2

Peering versus transit

Every price here is illustrative, not market data. Set how much traffic you exchange with one destination network and what the three options cost you. The chart plots the monthly cost of each option across traffic volumes on logarithmic axes; the dot marks your traffic, and the cheapest option there is ringed. The model assumes you add a port or link whenever one would pass 70% utilization, which is why the peering lines climb in steps.

Transit–
IXP peering–
Private interconnect–
Cheapest here–
IXP break-even–
IXP cost per Mb/s–

One address, many places

Anycast in depth

Anycast means announcing the same prefix from several places at once and letting BGP deliver each user to one of them. The DNS root servers and most large DNS and content services use it. The networks site shows the idea; here is how it actually behaves.

"Nearest" means nearest by BGP's rules, not by distance. Each network along the way picks one route to the anycast prefix using its own decision process: local preference first, which mostly encodes who pays whom, then AS path length, then tie-breakers such as the IGP cost to the exit. The set of users that end up at a given site is called its catchment, a word RFC 4786 borrows from river geography. A user in one city can be sent to a site on another continent because their provider's transit happens to reach that site in fewer AS hops, or because a peering link prefers it. Operators measure catchments; they do not assume them.

A host route cannot travel the internet. Networks commonly filter announcements longer than a /24 in IPv4 (Huston's 2025 report calls /24 the common minimum), so a single anycast address has to be announced inside a covering prefix, such as 203.0.113.0/24, and the whole block moves together.

A B ✕ C all three announce 203.0.113.0/24 withdrawn: health check failed dashed: users who moved when B withdrew
Catchments are decided by each network's BGP choice. When site B's health check fails and it withdraws the route, its users fall back to their next-best route, which may be any surviving site. The withdrawal is what makes failover work: a site that dies but keeps announcing becomes a black hole for its whole catchment.

RFC 4786, the best-current-practice document on operating anycast services, lists what can go wrong. The main hazards, in the order an operator meets them:

  • Announcing while broken. If the route stays up when the service is down, the site keeps attracting its catchment and drops it. RFC 4786 recommends coupling the announcement to service health, so a failed service withdraws its own route, and adding a delay before re-announcing so a flapping service does not cause route oscillation.
  • Paths that change mid-conversation. UDP queries such as DNS are one packet each way, so it barely matters which site answers. A TCP connection carries many packets, and if routing changes mid-flow, later packets arrive at a site that never saw the handshake, and the connection breaks. Long-lived sessions over anycast are a risk to plan for, not a free lunch.
  • Per-packet load balancing. A router that sprays packets of one flow across links toward different neighbors can split a single transaction across two sites. RFC 4786 calls networks doing this across different neighbor ASes, in many cases, "pathological."
  • Route flap dampening. Some networks suppress prefixes that change often. An anycast prefix that is withdrawn and re-announced at several sites can look like a flapping prefix and be suppressed everywhere at once.
  • Capacity and scope. A small site can be overwhelmed if its catchment turns out larger than expected. RFC 4786 describes local nodes, announced with the NO_EXPORT community (RFC 1997) so only nearby networks see them, alongside global nodes announced everywhere.
Go deeper: anycast and attacks

Anycast is also a defensive tool. A flood of traffic aimed at an anycast address is divided among the catchments of all its sites instead of landing on one, and a site that is overwhelmed can be withdrawn so its users drain elsewhere while the attack is absorbed by the rest. The same property makes it hard to tell, from one place, which site you are talking to; RFC 4786 discusses ways for clients and operators to identify the node that answered. Announcing someone else's prefix from an unauthorized place looks, to the routing system, exactly like adding an anycast site, which is why origin validation matters: see Chapter 11.

A million and counting

The global routing table

A router in the default-free zone has no default route: it carries an explicit route for every reachable prefix on the internet. The size of that table is one of the most-watched numbers in networking, because every entry costs memory and lookup work in every such router. Geoff Huston has published a report on it every January for many years; the figures below are his, rounded, measured at the start of each year from his own vantage point (other places in the network see slightly different numbers).

JanuaryIPv4 routesof which more-specificsIPv6 routesof which more-specificsIPv4 ASes
2022906,000486,000146,50088,70072,800
2023940,000495,000172,400103,00074,200
2024943,000486,000201,200117,20075,300
2025996,000526,000221,500127,50076,700
20261,050,000544,000241,800137,90077,900

Two things stand out. First, IPv4 is still growing, by about 53,000 entries between January 2024 and January 2025 and 54,000 in the following year, by the report's own table, even though the registries ran out of fresh IPv4 addresses years ago. Address trading and subdividing explain much of it: big blocks are being sliced up and sold, and each slice needs its own route. Second, more than half of the IPv4 table, 544,000 of 1,050,000 entries in January 2026, consists of more-specifics: prefixes that sit inside a larger prefix that is also announced. Every one of those is, in principle, redundant for reachability.

So why announce them? This is deaggregation, and the usual reasons are traffic engineering (announcing halves of a block through different providers to steer incoming traffic, the technique Chapter 9 covers), multihoming a small piece of a provider's block, and defending against hijacks of more-specifics, since the longest match wins. Huston questions how well the last one works in IPv6, where many networks accept prefixes much longer than the /48 that most deaggregators use. The size distribution is lopsided: /22, /23 and /24 prefixes together made up 84% of the IPv4 table at the end of 2025, and /48s about 45% of IPv6.

What does it cost? Each route sits in the router's RIB, with all its paths and attributes, in ordinary memory, and in the FIB, the forwarding table on each line card, often in expensive fast memory such as TCAM. Chapter 2 explains why a line rate lookup needs that hardware, and how tries compress the table. When a FIB fills, routers can start dropping routes or falling back to slow paths; operators plan hardware purchases around projections like Huston's.

Instrument 3

The table, year by year

Each point is a figure from Huston's January reports, as listed under Sources; nothing between the points is interpolated data, just a line to guide the eye. Tap or move across the chart to read a year. Switch views to split each year into root prefixes and more-specifics (available for 2022 to 2026 in the 2026 report).

January–
IPv4 routes–
IPv6 routes–
IPv4 growth that year–
Instrument 4

The deaggregation calculator

Pick a block you hold and the prefix length you choose to announce it as. Splitting a block of length a into prefixes of length b creates 2b−a routes, and every default-free router on the internet has to carry each one. The bar shows the block divided into its announced pieces.

Routes announced–
Addresses per route–
Share of the Jan 2026 table–
Will it propagate?–

/16 → /24s: 28 = 256 · /32 → /48s: 216 = 65,536
The two worked examples: an IPv4 /16 split into /24s, and an IPv6 /32 split into /48s. One IPv4 network splitting a /16 adds 255 routes to every full-table router; one IPv6 network splitting its /32 into /48s would add more than a quarter of the whole January 2026 IPv6 table by itself. Nothing in BGP stops it; filters and courtesy do.
Cheat sheet

Terms from this chapter

Autonomous system (AS)
A network, or group of networks, under one clearly defined routing policy, identified by an AS number and speaking BGP to its neighbors.
Four-byte ASN
An AS number in the 32-bit space introduced by RFC 6793; old two-byte routers see AS_TRANS (23456) in its place, with the true path carried in AS4_PATH.
Tier 1 (transit-free)
A network that reaches the whole internet through its customers and settlement-free peers without buying transit. A position in a web of contracts, not a size.
Stub AS
An AS that only originates its own prefixes and never carries traffic between others; most ASes are stubs.
Flattening
The shift of traffic onto direct links between content networks and access networks, bypassing the top of the old transit hierarchy.
Internet exchange point (IXP)
A shared switching fabric where many networks connect one port each and peer across a common LAN.
Route server
A BGP speaker at an IXP that redistributes members' routes to each other without forwarding traffic and without adding its own AS to the path (RFC 7947).
Private network interconnect (PNI)
A dedicated cross-connect between two networks' routers, used when they exchange enough traffic to fill whole ports.
PeeringDB
The public, user-maintained database of networks, exchanges and facilities that operators use to find peers.
Catchment
The set of users whose traffic BGP delivers to one particular site of an anycast service.
Default-free zone
The routers that carry an explicit route for every reachable prefix and no default route.
Deaggregation
Announcing a block as several longer prefixes instead of one; it steers traffic but grows every full routing table.
Where this comes from

Sources