Seeing the routing system

If traceroute and TTL are new, start with troubleshooting on networks.jelia.nyc, which has a traceroute simulator, and for what BGP is and why it carries policy, see dynamic routing. This chapter assumes both. Its question is narrower: given that you can't log in to anyone else's router, how do you find out what the internet's routing system is doing?

The main answer is the route collector. A collector is a BGP speaker that forwards no traffic at all. It sets up BGP sessions with networks that volunteer to feed it, accepts everything they send, announces nothing back, and writes every message to disk. Each network that feeds it is called a peer of the collector, and most peers send a full table: their own best path to every prefix they know. A collector is therefore a window onto many networks' routing decisions at once, as seen from each of their vantage points.

Two long-running public projects operate collectors. RouteViews, at the University of Oregon, describes itself as the oldest and most comprehensive public BGP archive, collecting from over a thousand peers across dozens of collectors, many of them at internet exchange points. RIPE RIS, the Routing Information Service of the RIPE NCC, runs collectors named RRC00 upward. Some sit on an exchange's peering LAN, where any member can peer with them directly; others are multihop collectors that accept BGP sessions across the internet from anywhere. Both projects publish their raw data free of charge.

A collector sees only what its peers choose to tell it, and that is a real limit. A network sends a collector its best path, not every path it holds, so backup routes are invisible. And because networks normally pass routes learned from a peer only to their customers (the "valley-free" pattern from chapter 09), many peering links between mid-sized networks never appear in any collector's view at all. The collectors show the internet's routing from a few hundred points of view. That is enormously useful, but it is not the whole picture. Chapter 10 covers the shape of the network those views are sampling.

Collectors store what they hear in MRT, the Multi-Threaded Routing Toolkit export format standardized in RFC 6396. An MRT file is a plain sequence of records, each with a small header saying when it was written and what kind of record follows. Two kinds dominate. A RIB dump is a snapshot of every route from every peer at one moment. An updates file holds the raw BGP UPDATE messages received in a short window, timestamped, so you can replay exactly what happened between snapshots. RIPE RIS currently writes a dump every 8 hours and an updates file every 5 minutes for each collector, under names like bview.YYYYMMDD.HHmm.gz and updates.YYYYMMDD.HHmm.gz. RouteViews publishes the same two kinds of file per collector in its archive.

Archives answer the question "what happened?" after the fact. For "what is happening now?", RIPE runs RIS Live, which streams the messages its collectors receive in near real time. Tools built on these feeds can raise an alarm within a minute or two of an unusual announcement.

Collectors listen to other people's routers. To watch your own, the standard tool is the BGP Monitoring Protocol, BMP, defined in RFC 7854. A BMP-capable router opens a TCP connection to a monitoring station and copies to it what its BGP sessions are receiving: the routes each neighbor sent before your import policy touched them (the pre-policy Adj-RIB-In), after it did (post-policy), or both. That difference is the point. With an ordinary BGP session you only see what a router decided to keep; with BMP you also see what it was offered and rejected. The RFC says the protocol exists to replace "screen scraping", logging in to routers and parsing command output, and that BMP is a one-way reporting channel, not a routing protocol.

Raw MRT is for programs. People usually start from tools that have already parsed the archives and can answer a question in one query. Each one answers a different question, and it pays to know which.

A typical investigation chains them. Users report that a service is unreachable from some countries. RIPEstat's routing status shows the prefix is seen by only part of the RIS peers, and the visibility history shows when that started. The paths from peers that still see it all pass through one upstream, and that upstream's neighbor list shows it lost a large transit provider at the same moment. PeeringDB tells you which exchanges that upstream is at and how to contact its operators. You have narrowed the problem to one network without touching anyone's equipment.

Every route a collector stores carries an AS_PATH: the list of autonomous systems the announcement passed through (chapter 08 shows the bytes). The most important rule is direction. The path reads from the collector's side on the left to the network that originated the prefix on the right, because each AS adds its own number to the front as it passes the route on. Traffic flows the opposite way, left to right, toward the origin.

Take 64496 64501 64500 64500 64500. AS64496 is the collector's peer, the network that told the collector. AS64500 is the origin, and AS64501 is its upstream, the network it sent the route to. AS64500 appears three times because it prepended: it added its own number extra times to make this path look longer, so that other networks would prefer a different way in (chapter 09 explains why). Prepending doesn't change who is on the path, so the first step in reading any path is to collapse the repeats. This one has five hops by BGP's count but crosses only three networks.

Some numbers should never appear on the public internet. Private ASNs, 64512–65534 and 4200000000–4294967294, are for use inside one organization, and RFC 6996 requires them to be removed before a route is advertised to the global internet; seeing one in a collector's path means somebody's filter is missing. AS 0 must never be sent at all (RFC 7607). The documentation ranges 64496–64511 and 65536–65551 exist for books and examples like this one. And AS_TRANS, 23456, is the placeholder a 4-byte-capable router puts in the path when it talks to an older router that can only handle 2-byte numbers (RFC 6793); the real number travels alongside in a separate attribute, so seeing 23456 in a modern path means it got lost on the way.

Finally, braces. A path ending in {64500,64502} contains an AS_SET, an unordered set left by route aggregation: several routes were merged and the router recorded every AS involved without an order. RFC 4271 counts a set as one hop. An AS_SET makes the origin ambiguous, which breaks origin validation, so the IETF first recommended against creating them and in 2025 deprecated them outright (RFC 9774).

Collectors show the control plane: what networks announce. Traceroute shows the data plane: where your packets actually went. The trick is old and simple. Send a probe with a time-to-live of 1; the first router decrements it to zero, drops the probe and sends back an ICMP Time Exceeded message, which reveals its address and how long the round trip took. Then send TTL 2, 3 and so on. The common Linux traceroute sends three UDP probes per TTL by default, to "unlikely" destination ports starting at 33434, until the destination answers with Port Unreachable or 30 hops pass.

Each line of the output is a separate experiment with a different router, and each answer is created by that router, about that router, at its own convenience. Five details follow from that, and each one has fooled experienced engineers.

1. The address is a choice. RFC 1812 says the source address of an ICMP message a router originates must belong to the interface the message is sent out of. If the reply leaves by a different interface than the one your probe arrived on, the address you see belongs to a link your probe never crossed, and looking it up can place the router in the wrong city or even the wrong network. Implementations differ here. RFC 5837 adds an object a router can attach to its reply stating outright which interface the probe arrived on.

2. A star at one hop is usually not loss. Forwarding happens in hardware at line rate; generating an ICMP error is work for the router's general-purpose processor, the slow path from chapter 01. RFC 1812 says routers should be able to limit how fast they send ICMP errors, and RFC 4443 makes rate limiting mandatory for ICMPv6 errors, recommending a token bucket. A router that is rate-limiting or busy answers some probes late or not at all, while forwarding everything else perfectly. The test is simple: if the later hops answer fully, nothing is wrong at that hop. Loss is only real if it continues all the way to the destination.

3. Several paths can hide in one trace. Routers split traffic across equal-cost paths by hashing each packet's addresses and ports (chapter 03). Classic traceroute changes the destination port for every probe, so each probe is a new flow and may be hashed onto a different branch. The result can be two addresses at the same TTL, or a "path" stitched together from pieces of different paths, with links between routers that are not actually connected. Brice Augustin, Timur Friedman and Renata Teixeira built Paris traceroute to fix this: it keeps the fields routers hash on constant for every probe, so all probes follow one path, and it can deliberately vary them to enumerate the branches.

4. MPLS can show or hide hops. Inside an MPLS core (chapter 12), a label-switching router whose TTL expires may not know how to reach your address at all. RFC 3032 describes the usual fix: send the Time Exceeded message onward along the label-switched path to its far end, which then routes it back. Every hop inside the tunnel therefore shows roughly the round-trip time of the tunnel's exit, not its own. RFC 4950 lets the router attach the label stack it received, which modern traceroutes print as "MPLS Label=…". An operator can also run tunnels in the "pipe" model, where the IP TTL isn't decremented inside the tunnel, so those routers never appear at all; RFC 4950 points out this TTL-transparent mode prevents Time Exceeded altogether.

5. The way back is invisible. Each RTT is the time to reach the hop plus the time for its reply to come home, and the reply follows the hop's own routing, not yours. Internet paths are often asymmetric, because each network chooses its own exit. So a 60 ms jump at hop 5 may be a long link between hops 4 and 5, or it may be that hop 5, and everything after it, sends its replies back across an ocean. A jump that persists to the destination is real for that destination, but traceroute alone cannot say which direction it lives in. To find out you need a measurement from the far side, from a looking glass or a measurement platform such as RIPE Atlas.

Because collectors see so many networks at once, they can show you problems with your own prefixes that your own routers never will. Your router will happily tell you it is announcing your prefix. Only an outside view can tell you whether anyone received it, and whether somebody else is announcing it too. Monitoring services built on RIS, RouteViews and similar feeds watch for four signals, and they are worth knowing whether you run such a service or simply read its alerts.

Visibility drops. The share of full-table collector peers that have any route to your prefix. A drop that settles at, say, 60% often means one of your upstreams stopped carrying the route, and the remaining 60% are networks that reach you through another one. A drop to zero is a withdrawal, deliberate or not. RIPEstat's visibility data counts peers against the set that send RIS a full table, because a peer that sends only its customers' routes was never going to show yours.

Unexpected origins. The same prefix announced by an AS that is not yours, a situation called MOAS (multiple origin AS). It can be legitimate, for example during a planned migration or with anycast, but it is also the signature of a hijack or a configuration typo elsewhere.

New more-specifics. A longer prefix inside yours, such as a /48 inside your /40. Because routers prefer the longest match (chapter 02), a more-specific attracts the traffic for its addresses everywhere it is accepted, whatever the path length. Unexpected more-specifics deserve the fastest alert.

New upstreams or new paths. A change in which AS appears just before yours, or a sudden path through networks that have no business carrying your traffic, can point to a route leak. It can also be a new contract your colleagues forgot to mention, which is why good monitoring lets you mark changes as expected.

The defences against these events, route origin validation with RPKI, ASPA and filtering, belong to chapter 11. Monitoring is the part that tells you, within minutes, that you need them. RIPE's routing beacons, prefixes announced and withdrawn on a fixed two-hour schedule, are a public calibration signal for this work: because the timing of each change is known in advance, you can measure how long news takes to reach each collector peer.

Almost everything in this chapter uses data that networks have chosen to publish, and that is how it should stay. Some plain rules, which the operators of these services also ask for:

• Prefer the archives. Collector data already holds the control-plane view of thousands of networks. Answering a question from MRT files or RIPEstat costs nobody anything.
• Respect looking glasses. They run on production routers. Follow each one's terms, don't script them, and use one query where one will do.
• Trace your own paths, gently. A traceroute to a service you use, a few probes per hop, is ordinary troubleshooting. Sweeping traceroutes or scans across someone else's address space is not, and it is exactly the traffic that triggers the rate limits above.
• Use measurement platforms for scale. If you need measurements from many places, platforms such as RIPE Atlas exist so that probes are coordinated and accounted for.
• Report, don't exploit. If you find a leaked private ASN or a stray announcement, tell the network through its PeeringDB or registry contacts.