Traceroute Enrichment

A raw traceroute hop is an IP address and some round-trip times. On its own that cannot answer “which network was this, and where was it?” — the questions localization depends on. This page describes how the traceroute enrichment pipeline turns each hop into an annotated hop, which sources it uses, how it records where each annotation came from, and how it flags annotations that are probably wrong.

The traceroute enrichment pipeline is part of HERMES. It produces the annotated paths that HERMES uses as evidence when localizing a performance degradation. For the HERMES system, see the HERMES overview; for the stage’s place in the pipeline, see the methodology.

If traceroutes are unfamiliar, read Introduction to traceroutes first.

Inputs

HERMES enriches two path measurements per NDT test, where both are available.

The path from the M-Lab server toward the client comes from M-Lab’s own traceroute core service. For every TCP connection to an M-Lab server, traceroute-caller runs scamper back toward the client and publishes the result as the scamper1 datatype. HERMES consumes this, together with M-Lab’s hopannotation data. Where no scamper1 trace exists for a measurement, as on bring-your-own-server (BYOS) deployments, HERMES uses the standard traceroute those servers publish in the scamper2 format instead.

The path from the client back toward the server comes from reverse traceroute. Reverse paths cannot be measured directly from the server, so they are inferred using vantage points and IP options. This is an inference, and HERMES keeps the reverse-traceroute system’s own metadata about it in the row: which system produced the path, why the measurement stopped, why it failed where it did, whether a vantage point in the client’s AS was attempted, and per-hop, whether the hop relies on an assumption of interdomain path symmetry.

Note that “forward” and “reverse” in traceroute vocabulary do not describe the endpoints. The published schema names the directions as measured — server_to_client_path and client_to_server_path — and preserves each path’s measured hop order and RTT vantage point rather than reversing arrays and implying RTTs originated somewhere they did not.

What gets attached to a hop

Every geographic annotation carries its source and a confidence score alongside it. That pairing is deliberate: a hop location is an inference, and a dataset that presents inferences without their provenance invites them to be read as observations.

Data sources

Two distinct mappings place a hop in the network: one puts the address in an autonomous system, the other recognises it as an interconnection fabric. They come from different sources and fail in different ways, so they are worth keeping apart when reading a path.

The lookup tables above live in the mlab-collaboration.hermes dataset, which is currently available on request: email support@measurementlab.net. The annotated paths themselves are in mlab-collaboration.hermes_union.events_enriched, which members of the M-Lab Discuss group can query.

Because the AS and IXP mappings are independent, a hop can carry one without the other: an address may resolve to an AS with no IXP match, or sit in an IXP peering LAN whose address the prefix mapping attributes to the IXP operator rather than to either peer. Read asn and ixp together rather than treating one as a fallback for the other.

Client geolocation moved from MaxMind to IPinfo on 2026-08-01; backfilled dates through 2025-07-31 also use IPinfo. Rows carry client.geo_source so you can tell which source placed a given client. See the schema changelog.

Trustworthiness checks

Enrichment can be wrong. These checks exist so that consumers can tell when it probably is, rather than having to assume it never is.

Speed of light

Signals in fiber propagate at roughly 200,000 km/s — about two thirds of the speed of light in vacuum. This bounds how quickly a reply can come back from a given distance.

For each hop, HERMES records the assumed propagation speed (propagation_speed_km_s) and the resulting lower bound on RTT (fiber_lower_bound_rtt_ms), then compares that bound against the RTT actually measured. distance_rtt_check reports the result.

When a measured RTT is lower than physics allows for the geolocated distance, the physics is not wrong: the geolocation is. This is the single most useful signal for spotting a bad hop location.

Distance accumulation

segment_distance_km is the distance from the previous geolocated hop, cumulative_distance_km the distance travelled so far in measured order, and remaining_distance_km the distance still to go. Together with the endpoints’ straight-line separation, these give the path’s detour_ratio.

A detour ratio well above 1 means traffic travelled substantially further than it needed to — often the visible signature of a rerouting event. Read it alongside geolocation_coverage: a distance computed from a path where only a third of hops could be placed is a lower bound assembled from the hops that happened to be geolocated, not a measurement of the route.

Latency behaviour along the path

above_baseline_flag and increasing_latency_flag describe how each hop’s latency behaves relative to the path’s own baseline, and baseline_consistency_flag summarizes whether the path as a whole is consistent with its baseline. These distinguish a path that is uniformly slower from one where latency jumps at a specific hop.

Path-level flags

Carried on the path records rather than per hop:

• loop_detected — an AS reappears after a different AS (A B A). This can be a real routing loop, but it is also the signature of third-party addresses and mis-mapped hops, so treat it as a reason for caution rather than a diagnosis. Staying in one AS for several hops is normal and is not a loop.
• unresponsive_within_as — an AS reappears after hops with no ASN (A … A): part of that AS’s internal path did not reply or could not be mapped, so it is invisible.
• reaches_client / reaches_client_asn — whether the measurement got all the way to the client, or only as far as its AS.
• is_virtual — the path shows signs of being a virtual or tunnelled route rather than the physical topology.
• geolocation_coverage — what fraction of hops could be geolocated at all.

Filter on these carefully before drawing conclusions from any traceroute analysis. A path that never reached the client cannot support a statement about the last mile.

Limitations

These are properties of traceroute and of IP annotation generally. They bound what any path-based localization can conclude.

Unresponsive hops. Routers may be configured not to reply, or to rate-limit replies. A gap in a path is not an absence of infrastructure; it is an absence of information. responsive_hop_count and total_hop_count tell you how much of the path you are actually seeing.

Third-party addresses. A router may reply from an interface that belongs to a different network than the one carrying the traffic, which can attribute a hop to the wrong AS and place it in the wrong location. This is a well-known source of error in AS-level path inference from traceroute.

Prefix mapping gives the prefix’s origin AS, not the interface’s operator. asn comes from a longest-match lookup in a BGP-derived IP-to-prefix table. That answers “which AS originates the prefix containing this address”, which is usually but not always the network operating the router that replied — point-to-point links, address space lent between networks, and the third-party case above all break the equivalence.

Geolocation error. IP geolocation is inference from allocation records, hostnames, and latency, not observation. Error at the city level is common, and infrastructure addresses are harder to place than residential ones. geo_score and distance_rtt_check are the fields to consult; geo_source tells you which method was responsible.

Load balancing. Consecutive probes may take different paths through a load-balanced network, so a single traceroute is one sample of a set of possible routes.

A hop’s RTT is not a link’s latency. RTT to a hop is a round trip that includes the return path from that hop, which may differ entirely from the path back from the destination. Rising RTT at a hop does not localize a problem to that hop by itself.

Source

The enrichment stage is implemented in 04_mapping_union.sql and the hermes.enrichment package in github.com/m-lab/hermes.