<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.9.0">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2026-03-09T11:59:33+00:00</updated><id>/feed.xml</id><title type="html">M-Lab</title><entry><title type="html">M-Lab and Giga to Launch Community of Practice to Advance School Connectivity Measurement</title><link href="/blog/giga-mlab-cop/" rel="alternate" type="text/html" title="M-Lab and Giga to Launch Community of Practice to Advance School Connectivity Measurement" /><published>2026-02-26T00:00:00+00:00</published><updated>2026-02-26T00:00:00+00:00</updated><id>/blog/giga-mlab-cop</id><content type="html" xml:base="/blog/giga-mlab-cop/">&lt;p&gt;&lt;img src=&quot;/images/blog/2026-02-26-giga-mlab-cop/giga_x_m-lab.jpg&quot; alt=&quot;Giga M-Lab&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;M-Lab and Giga to Launch Community of Practice to Advance School Connectivity Measurement&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Measurement Lab (M-Lab) is excited to announce the early launch of a new Connectivity Community of Practice for applied Internet measurement in collaboration with Giga.&lt;!--more--&gt;&lt;/p&gt;

&lt;p&gt;As the world’s largest open collection of Ιnternet performance data, M-Lab provides powerful, real-world telemetry that helps illuminate how networks actually perform. Through this collaboration, we’re working with Giga to strengthen how connectivity is measured and understood for public facilities, especially schools, using open measurement approaches that reflect on-the-ground conditions.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://giga.global/&quot;&gt;Giga&lt;/a&gt; is a joint initiative of UNICEF and the International Telecommunication Union (ITU), working to connect every school to the Internet and every young person to information, opportunity, and choice. Through its global focus on school connectivity, Giga supports governments and partners with data, technical expertise, and financing tools to accelerate meaningful access to the Internet for education.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Objectives&lt;/strong&gt;&lt;br /&gt;
This Community of Practice (CoP) builds on that shared foundation. It brings together researchers, network engineers, implementers, and policymakers to advance measurement approaches that are:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Technically rigorous&lt;/li&gt;
  &lt;li&gt;Globally comparable&lt;/li&gt;
  &lt;li&gt;Practical for real-world connectivity contexts, including underserved regions&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Together, the community will first explore what meaningful measurement looks like for schools, align on indicators that support better decision-making, and translate open telemetry into insights that enable smarter, evidence-based connectivity planning.&lt;/p&gt;

&lt;p&gt;At its core, the CoP reflects a shared commitment to openness in data, in tools, and in collaboration. As the community grows, we look forward to shaping inclusive participation and governance models with contributors, ensuring this work remains durable, representative, and grounded in shared priorities.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Join us!&lt;/strong&gt;&lt;br /&gt;
We’ll host a virtual launch event on March 11, 2026 at 14:00 CET / 9:00 Eastern Time  to introduce the Community of Practice’s initial focus areas and invite input into its first-year agenda.&lt;/p&gt;

&lt;p&gt;Whether you’re an engineer, researcher, educator, advocate, or policymaker, we welcome everyone working at the intersection of Ιnternet measurement and school connectivity to join us and help shape what comes next. If you’re interested in joining the CoP, &lt;em&gt;click on the registration link below&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://us02web.zoom.us/meeting/register/Md7jAQrcQo2iKuPusBiXpQ&quot;&gt;&lt;strong&gt;Register to attend here&lt;/strong&gt;&lt;/a&gt;!&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Can’t attend live? Join the &lt;a href=&quot;https://forms.gle/rU9o5vuT4mmzYE7p7&quot;&gt;CoP Connectivity group&lt;/a&gt; to receive event materials and updates on future activities.&lt;/em&gt;&lt;/p&gt;</content><author><name>Melissa Newcomb, Pavlos Sermpezis</name></author><category term="event" /><category term="community" /><category term="announcement" /><category term="partnership" /><summary type="html">M-Lab and Giga to Launch Community of Practice to Advance School Connectivity Measurement Measurement Lab (M-Lab) is excited to announce the early launch of a new Connectivity Community of Practice for applied Internet measurement in collaboration with Giga.</summary></entry><entry><title type="html">OMG! The Fourth Open Measurement Gathering Report is Out Now</title><link href="/blog/omg4/" rel="alternate" type="text/html" title="OMG! The Fourth Open Measurement Gathering Report is Out Now" /><published>2025-12-22T00:00:00+00:00</published><updated>2025-12-22T00:00:00+00:00</updated><id>/blog/omg4</id><content type="html" xml:base="/blog/omg4/">&lt;p&gt;&lt;strong&gt;The Open Measurement Gatherings&lt;/strong&gt;&lt;br /&gt;
From 2024 to 2025, &lt;a href=&quot;https://www.measurementlab.net/&quot;&gt;Measurement Lab&lt;/a&gt;, &lt;a href=&quot;https://censoredplanet.org/&quot;&gt;Censored Planet&lt;/a&gt;, &lt;a href=&quot;https://ioda.inetintel.cc.gatech.edu/&quot;&gt;IODA&lt;/a&gt;, and &lt;a href=&quot;https://ooni.org/&quot;&gt;OONI&lt;/a&gt; organized four convenings called the Open Measurement Gatherings (OMG). The intention of these convenings was to build understanding, trust, and collaboration across open Internet measurement groups and this initiative was made possible with the support of the &lt;a href=&quot;https://www.opentech.fund/&quot;&gt;Open Technology Fund&lt;/a&gt; (OTF).&lt;/p&gt;

&lt;p&gt;From September 8-12, 2025, the Open Measurement Groups groups gathered in Estoril, Portugal for the &lt;a href=&quot;/publications/OMG4_final_report.pdf&quot;&gt;fourth convening of the two-year program&lt;/a&gt;.&lt;!--more--&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-12-22-omg4/OMG4_group.png&quot; alt=&quot;OMG4 group&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Open Measurement Village&lt;/strong&gt; &lt;br /&gt;
From Sept 8-10, 2025, OMG groups hosted an Open Measurement Village at the &lt;a href=&quot;https://gathering.digitalrights.community/&quot;&gt;Global Gathering&lt;/a&gt; (GG). At the Village, OMG groups took turns hosting a booth, circle discussions, and a game called “Are You My Measurement Tool?” to help compare the use cases and data of each OMG group’s measurement tool. Balancing innovation in Internet measurement with accessibility is a key challenge that came up throughout the Global Gathering. Open measurement groups must keep up with increasingly sophisticated Internet interference while educating the non-technical community.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-12-22-omg4/OMG_game.png&quot; alt=&quot;OMG4 game&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;New OMG User Guide&lt;/strong&gt;&lt;br /&gt;
Based on the feedback from the GG community, the OMG groups created the &lt;a href=&quot;/publications/OMG4_User_Guide.pdf&quot;&gt;OMG User Guide&lt;/a&gt; as a public resource to help users determine which measurement tool they should use or how to compare data from OMG groups.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-12-22-omg4/OMG_User_Guide.png&quot; alt=&quot;OMG4 user guide&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;OMG 4&lt;/strong&gt; &lt;br /&gt;
The OMG groups remained in Estoril from September 11-12 to reflect on community feedback and the previous OMG convenings, share updates, and strategize on how to continue the momentum to deepen collaboration following the completion of the OTF-funded program. OMG groups face shared challenges; to counter those challenges we need to increase coordination, and to continue the work the open measurement groups must become sustainable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Impact of the OMG Convenings&lt;/strong&gt; &lt;br /&gt;
The OMG convenings deepened mutual understanding among the groups about each platform, tools, data pipelines, and users. Most importantly though, these convenings deepened the interpersonal relationships across OMG groups and built trust so that OMG groups can hold joint public events, preview upcoming changes, solve problems together, and strategize for the future of open measurement to meet a moment of unprecedented challenges.&lt;/p&gt;

&lt;p&gt;These convenings developed a default habit to collaborate, incorporate data across groups, and build collective awareness of each group with our users. Going forward, the OMG groups have several actionable, short and long term plans to continue collaboration. Despite reduced resources, open measurement matters now more than ever given current global trends in policy and technology.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;OMG 4 Report&lt;/strong&gt;&lt;br /&gt;
For all the details about the Open Measurement Village, the Open Measurement Gathering, and our plans going forward, check out the &lt;a href=&quot;/publications/OMG4_final_report.pdf&quot;&gt;full report&lt;/a&gt;!&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-12-22-omg4/OMG_Logos.png&quot; alt=&quot;OMG4 logos&quot; /&gt;&lt;/p&gt;</content><author><name>Melissa Newcomb, Pavlos Sermpezis</name></author><category term="event" /><category term="censorship measurement" /><category term="community" /><summary type="html">The Open Measurement Gatherings From 2024 to 2025, Measurement Lab, Censored Planet, IODA, and OONI organized four convenings called the Open Measurement Gatherings (OMG). The intention of these convenings was to build understanding, trust, and collaboration across open Internet measurement groups and this initiative was made possible with the support of the Open Technology Fund (OTF). From September 8-12, 2025, the Open Measurement Groups groups gathered in Estoril, Portugal for the fourth convening of the two-year program.</summary></entry><entry><title type="html">M-Lab at Pulse Research Week with Internet Society and Giga</title><link href="/blog/pulse-research-week/" rel="alternate" type="text/html" title="M-Lab at Pulse Research Week with Internet Society and Giga" /><published>2025-12-18T00:00:00+00:00</published><updated>2025-12-18T00:00:00+00:00</updated><id>/blog/pulse-research-week</id><content type="html" xml:base="/blog/pulse-research-week/">&lt;p&gt;From December 8-11, 2025,  Measurement Lab (M-Lab) joined the Internet Society
&lt;a href=&quot;https://www.internetsociety.org/events/internet-society-pulse-research-week/&quot;&gt;Pulse Research
Week&lt;/a&gt;.
On Day 4 Pavlos Sermpezis, M-Lab’s Director, Technical Lead, presented the
&lt;a href=&quot;https://www.measurementlab.net/blog/iqb/#measurement-lab-publishes-the-internet-quality-barometer-framework&quot;&gt;Internet Quality
Barometer&lt;/a&gt;
prototype at the &lt;a href=&quot;https://giga.global/&quot;&gt;Giga&lt;/a&gt; offices in Barcelona. M-Lab
discussed with the audience how IQB could be adapted for education.&lt;!--more--&gt;&lt;/p&gt;

&lt;p&gt;During the closing session, Melissa Newcomb, M-Lab’s Senior Program Manager,
led a reflection session featuring Georgia Bullen (M-Lab Advisory Committee
Chair), Steve Song (Internet Society), and Shilpa Arora (Giga). The panel
discussed how to take the knowledge gathering throughout the week and how to
apply it to solve challenges in connectivity.&lt;/p&gt;

&lt;p&gt;Throughout Pulse Research Week, the M-Lab team learned from diverse
stakeholders and experts who are passionate about improving connectivity
worldwide. The Internet Society and Giga are critical partners for M-Lab and it
was powerful to combine our shared expertise and apply it to the issue of
improving connectivity in underserved regions. We look forward to carrying the
momentum of last week into 2026!&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-12-18-pulse-research-week/image1.jpg&quot; alt=&quot;2025 Pulse Research Week #1&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-12-18-pulse-research-week/image2.jpg&quot; alt=&quot;2025 Pulse Research Week #2&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-12-18-pulse-research-week/image3.jpg&quot; alt=&quot;2025 Pulse Research Week #3&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-12-18-pulse-research-week/image4.jpg&quot; alt=&quot;2025 Pulse Research Week #4&quot; /&gt;&lt;/p&gt;</content><author><name>Melissa Newcomb, Pavlos Sermpezis</name></author><category term="event" /><category term="community" /><category term="research" /><category term="data" /><summary type="html">From December 8-11, 2025, Measurement Lab (M-Lab) joined the Internet Society Pulse Research Week. On Day 4 Pavlos Sermpezis, M-Lab’s Director, Technical Lead, presented the Internet Quality Barometer prototype at the Giga offices in Barcelona. M-Lab discussed with the audience how IQB could be adapted for education.</summary></entry><entry><title type="html">A Hands-On Tutorial with Reverse Traceroute</title><link href="/blog/revtr_tutorial/" rel="alternate" type="text/html" title="A Hands-On Tutorial with Reverse Traceroute" /><published>2025-11-13T00:00:00+00:00</published><updated>2025-11-13T00:00:00+00:00</updated><id>/blog/revtr_tutorial</id><content type="html" xml:base="/blog/revtr_tutorial/">&lt;p&gt;Internet paths are often asymmetric: packets from A to B usually take a different route than packets in the reverse direction from B to A. This post walks through a reproducible case study using &lt;strong&gt;Reverse Traceroute (RevTr)&lt;/strong&gt; to compare forward and reverse paths for NDT speed tests.&lt;!--more--&gt; We’ll show how to:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Select a test where both the forward and reverse traceroutes reached their destinations.&lt;/li&gt;
  &lt;li&gt;Pull the data directly from BigQuery.&lt;/li&gt;
  &lt;li&gt;Visualize where and how the two directions differ.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This post focuses on the workflow: how to pull tests, line up forward and reverse traceroutes, and compare them. If you want a more detailed walk-through of the Reverse Traceroute output itself: what each entry in the table means, how to interpret its output, and how to understand the details of the technique, please refer to the &lt;a href=&quot;https://www.measurementlab.net/tests/reverse_traceroute/&quot;&gt;detailed documentation of the reverse traceroute dataset&lt;/a&gt;.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;why-do-forward-and-reverse-paths-differ&quot;&gt;&lt;strong&gt;Why do forward and reverse paths differ?&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;Internet routing is &lt;strong&gt;policy-driven&lt;/strong&gt;. Business relationships, peering choices, and traffic-engineering decisions can send return traffic through a completely different set of networks or cities than the forward path. For example, traffic from a device in NYC to a device in LA may flow through Chicago and Denver, while traffic on the way back from LA to NYC may flow through Dallas and Atlanta. To properly diagnose performance problems, it’s essential to see &lt;strong&gt;both directions&lt;/strong&gt;.&lt;/p&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;data-sources&quot;&gt;&lt;strong&gt;Data sources&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;In this case study, we’ll use:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;NDT speed tests&lt;/strong&gt; collected by M-Lab.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Reverse paths&lt;/strong&gt; (client → server) from the RevTr system, stored in BigQuery at&lt;br /&gt;
 &lt;em&gt;measurement-lab.revtr_raw.revtr1.&lt;/em&gt;&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Forward traceroutes&lt;/strong&gt; (server → client) from&lt;br /&gt;
 &lt;em&gt;Measurement-lab.revtr_raw.trace1.&lt;/em&gt;&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Hop annotations&lt;/strong&gt; collected by M-Lab.&lt;/li&gt;
&lt;/ul&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;what-the-notebook-does&quot;&gt;&lt;strong&gt;What the notebook does&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;Not every NDT test has a reverse traceroute — in fact, only about 25% do. We’ll focus on that subset and:&lt;/p&gt;

&lt;ol&gt;
  &lt;li&gt;Pick a day and select tests where both forward and reverse paths reach their destinations.&lt;/li&gt;
  &lt;li&gt;Extract hop-by-hop IP addresses and map to AS numbers, geolocation, and RTT from the source.&lt;/li&gt;
  &lt;li&gt;Compare forward and reverse directions by looking at:
    &lt;ul&gt;
      &lt;li&gt;&lt;strong&gt;AS-path overlap&lt;/strong&gt; (common subsequences).&lt;/li&gt;
      &lt;li&gt;&lt;strong&gt;Geographic path length&lt;/strong&gt; (sum of great-circle distances).&lt;/li&gt;
      &lt;li&gt;&lt;strong&gt;Map overlays&lt;/strong&gt; to see where the paths diverge.&lt;/li&gt;
    &lt;/ul&gt;
  &lt;/li&gt;
&lt;/ol&gt;

&lt;hr /&gt;

&lt;h2 id=&quot;what-to-look-for&quot;&gt;&lt;strong&gt;What to look for&lt;/strong&gt;&lt;/h2&gt;

&lt;p&gt;When you visualize the two directions, ask:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Where do the AS paths diverge?&lt;/li&gt;
  &lt;li&gt;Does one direction cross different cities than the other?&lt;/li&gt;
  &lt;li&gt;Is one direction significantly longer in kilometers or showing higher RTT growth?&lt;/li&gt;
  &lt;li&gt;Do you notice any suspicious geographic loops that could indicate policy quirks?&lt;/li&gt;
&lt;/ul&gt;

&lt;hr /&gt;

&lt;h3 id=&quot;try-it-yourself&quot;&gt;&lt;strong&gt;Try it yourself&lt;/strong&gt;&lt;/h3&gt;

&lt;p&gt;The notebook cells below are ready to run:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;It will find a candidate test with both directions.&lt;/li&gt;
  &lt;li&gt;Fetch and parse the paths.&lt;/li&gt;
  &lt;li&gt;Compute overlap metrics.&lt;/li&gt;
  &lt;li&gt;Generate simple visualizations (AS-level, geographic, and RTT-based).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If the first run does not yield an interesting pair of paths, simply change the index (or the date) and try again.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://colab.research.google.com/drive/12KET9GAtRJU0ZSGGxikxrVNF3wzAUcRL?usp=sharing&quot;&gt;RevTr_UseCase.ipynb&lt;/a&gt;&lt;/p&gt;</content><author><name>Loqman Salamatian</name></author><category term="data" /><category term="community" /><category term="research" /><category term="bigquery" /><category term="traceroute" /><summary type="html">Internet paths are often asymmetric: packets from A to B usually take a different route than packets in the reverse direction from B to A. This post walks through a reproducible case study using Reverse Traceroute (RevTr) to compare forward and reverse paths for NDT speed tests.</summary></entry><entry><title type="html">Open Measurement Hackathon at the ACM Internet Measurement Conference (IMC) 2025</title><link href="/blog/imc2025-hackathon/" rel="alternate" type="text/html" title="Open Measurement Hackathon at the ACM Internet Measurement Conference (IMC) 2025" /><published>2025-11-12T00:00:00+00:00</published><updated>2025-11-12T00:00:00+00:00</updated><id>/blog/imc2025-hackathon</id><content type="html" xml:base="/blog/imc2025-hackathon/">&lt;p&gt;M-Lab, &lt;a href=&quot;https://ooni.org/&quot;&gt;OONI&lt;/a&gt;, &lt;a href=&quot;https://www.iij.ad.jp/en/&quot;&gt;IIJ&lt;/a&gt;, &lt;a href=&quot;https://dioptra.io/&quot;&gt;Dioptra Research Group&lt;/a&gt;, &lt;a href=&quot;https://radar.cloudflare.com/&quot;&gt;Cloudflare Radar&lt;/a&gt;, &lt;a href=&quot;https://www.measurementlab.net/tests/reverse_traceroute/&quot;&gt;Reverse Traceroute&lt;/a&gt;, and &lt;a href=&quot;https://ioda.inetintel.cc.gatech.edu/&quot;&gt;IODA&lt;/a&gt; hosted the &lt;a href=&quot;https://conferences.sigcomm.org/imc/2025/events/hackathon/&quot;&gt;Open Measurement Hackathon&lt;/a&gt;, October 31, 2025, following the &lt;a href=&quot;https://conferences.sigcomm.org/imc/2025/&quot;&gt;ACM Internet Measurement Conference (IMC) 2025&lt;/a&gt; in Madison, Wisconsin, USA.&lt;!--more--&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-11-12-imc2025-hackathon/imc2025_hackathon_logo.png&quot; alt=&quot;IMC 2025 hackathon logo&quot; style=&quot;width: 40%; height: auto;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Over 40 participants from academia and industry gathered to &lt;em&gt;“Measure All the Frights”&lt;/em&gt;, exploring and comparing data from our projects over the eight-hour hackathon. Each host organization suggested potential projects to participants in advance, and those who planned to attend formed teams via Discord or in person to select a project or propose their own ideas. Teams worked on a variety of projects, including:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;Aggregating and visualizing metrics across multiple platforms/datasets&lt;/li&gt;
  &lt;li&gt;Analyzing and visualizing internet censorship data&lt;/li&gt;
  &lt;li&gt;Examining encrypted client hello (ECH) blocking&lt;/li&gt;
  &lt;li&gt;Examining TLS certificate diversity across countries/networks&lt;/li&gt;
  &lt;li&gt;Exploring aggregated country level internet performance data&lt;/li&gt;
  &lt;li&gt;Reproducing and extending key results from a recent study on web centralization&lt;/li&gt;
  &lt;li&gt;Comparing IPv4 versus IPv6 performance&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The event concluded with teams presenting their findings to the group. The &lt;a href=&quot;https://github.com/m-lab/open-measurement-hackathon&quot;&gt;&lt;strong&gt;hackathon GitHub repository&lt;/strong&gt;&lt;/a&gt; lists the projects that teams worked on at the event as well as code, presentations, images, etc. that participants shared with the hackathon organizers.&lt;/p&gt;

&lt;p&gt;We’re grateful for everyone who brought their energy and ideas to the IMC 2025 hackathon. We enjoyed meeting you, talking about your research, and appreciate your interest in our open source and open data projects! Thanks as well to &lt;a href=&quot;https://www.internetsociety.org/&quot;&gt;Internet Society&lt;/a&gt; for sponsoring the venue, beverages, and dinner for those who attended.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-11-12-imc2025-hackathon/imc_2025_hackathon_photo2.jpg&quot; alt=&quot;IMC 2025 hackathon logo&quot; /&gt;&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-11-12-imc2025-hackathon/imc_2025_hackathon_photo1.png&quot; alt=&quot;IMC 2025 hackathon logo&quot; /&gt;&lt;/p&gt;</content><author><name>Chris Ritzo, Melissa Newcomb, Pavlos Sermpezis</name></author><category term="event" /><category term="community" /><category term="research" /><category term="data" /><summary type="html">M-Lab, OONI, IIJ, Dioptra Research Group, Cloudflare Radar, Reverse Traceroute, and IODA hosted the Open Measurement Hackathon, October 31, 2025, following the ACM Internet Measurement Conference (IMC) 2025 in Madison, Wisconsin, USA.</summary></entry><entry><title type="html">Measurement Lab publishes new dataset of NDT7 measurements</title><link href="/blog/dynamic-data-bq/" rel="alternate" type="text/html" title="Measurement Lab publishes new dataset of NDT7 measurements" /><published>2025-07-15T00:00:00+00:00</published><updated>2025-07-15T00:00:00+00:00</updated><id>/blog/dynamic-data-bq</id><content type="html" xml:base="/blog/dynamic-data-bq/">&lt;p&gt;A new dataset of ndt7 measurements is now available in BigQuery, offering access to measurements from a set of servers whose data had not been previously published. &lt;!--more--&gt;&lt;/p&gt;

&lt;div style=&quot;background-color: #E7F1FF; padding: 10px;&quot;&gt;
  These servers and their corresponding datasets are still in the experimental phase. We invite users to explore this new dataset and provide feedback. If you encounter any issues or notice any inconsistencies, please let us know so that we can continue to improve the quality and reliability of this dataset.
&lt;/div&gt;
&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Where do these data come from?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Measurement Lab is in the second pilot phase to test a new dynamic method for deploying and managing measurement servers.  As mentioned in our &lt;a href=&quot;https://www.youtube.com/watch?v=da3zinXuqnY&quot;&gt;Community Call in November 2024&lt;/a&gt;, M-Lab’s platform is evolving to better measure the Internet today, by enabling the addition of new servers to the M-Lab fleet through the &lt;a href=&quot;https://github.com/m-lab/autonode/wiki/Host%E2%80%90managed-Deployments&quot;&gt;Host Managed Deployment Program&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://www.measurementlab.net/status/&quot;&gt;M-Lab’s platform&lt;/a&gt; consists of more than 400 servers in more than 40 countries and 100 metros. The majority of these servers belong to the “legacy” fleet, whose measurements are already available in &lt;a href=&quot;https://console.cloud.google.com/bigquery?project=measurement-lab&quot;&gt;BiqQuery&lt;/a&gt;, e.g., in the table `&lt;code class=&quot;language-plaintext highlighter-rouge&quot;&gt;measurement-lab.ndt.ndt7&lt;/code&gt;`.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-07-17_dynamic-data-bq/mlab-servers-legacy-july2025.png&quot; alt=&quot;M-Lab servers legacy fleet&quot; style=&quot;width: 80%; height: auto;&quot; /&gt; 
&lt;strong&gt;Figure 1&lt;/strong&gt;: The “legacy” fleet of M-Lab servers. It consists of more than 370 servers.&lt;/p&gt;

&lt;p&gt;A new set of servers, which we call as the &lt;em&gt;dynamic&lt;sup id=&quot;fnref:1&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:1&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt;&lt;/em&gt; fleet of servers–including those deployed through the Host Managed Deployment Program–has been gradually rolled out as part of a pilot phase. Although data collection from these servers has been ongoing for the past year, these data have not been publicly available until now.&lt;/p&gt;

&lt;p&gt;&lt;img src=&quot;/images/blog/2025-07-17_dynamic-data-bq/mlab-servers-dynamic-july2025.png&quot; alt=&quot;M-Lab servers dynamic fleet&quot; style=&quot;width: 80%; height: auto;&quot; /&gt;&lt;br /&gt;
&lt;strong&gt;Figure 2&lt;/strong&gt;: The “dynamic” fleet of M-Lab servers as of July 2025. It consists of more than 35 servers in 9 countries.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Accessing the new data in BigQuery&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Until recently, data from the dynamic fleet was withheld to ensure integrity while we piloted the dynamic deployment system. Now, we would like to make the data available. However, we are still in an experimental stage and maintain separate tables to make it easy for researchers to see how the data is changing.&lt;/p&gt;

&lt;p&gt;There have been no changes to NDT itself, but the BigQuery schemas are slightly different due to changes in server metadata and the methods used to import the data into BigQuery.&lt;/p&gt;

&lt;p&gt;Specifically, to accommodate these schema differences, we’re publishing three new views in the measurement-lab.ndt dataset:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;ndt.ndt7_legacy&lt;/strong&gt; - The full legacy ndt7 view as currently published as ndt.ndt7.  This view will be preserved to support investigating the provenance of the legacy data.  It supports locating the raw data that was processed in order to populate BigQuery.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;ndt.ndt7_dynamic&lt;/strong&gt; - Fully annotated data from the dynamically registered fleet, including the provenance of the data, so researchers can locate the raw data in the archive.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;ndt.ndt7_union&lt;/strong&gt; - The union of ndt7_legacy and ndt7_dynamic, but without provenance because it is incompatible between the sources.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Notes for researchers and future plans&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Most researchers using ndt7 data won’t need to take any action. Your existing queries using ndt.ndt7 should continue to work seamlessly through this transition.&lt;/p&gt;

&lt;p&gt;If you would like to explore all data (both from legacy and dynamic fleets), we recommend temporarily switching to ndt7_union.  Researchers using server metadata such as server DNS names, IP addresses, routing information and Anonymous System Numbers (ASNs) should still be able to parse the data, possibly with a few minor changes to generalize regular expressions. (Site names may now include more than two digits, e.g., abc1234).&lt;/p&gt;

&lt;p&gt;Currently, the dynamic fleet of servers provides tests and data only for ndt7; in the future, it will support measurements for other tools (reverse traceroute, WeHe, Neubot, etc).&lt;/p&gt;

&lt;p&gt;At some point in the future we will redeploy ndt.ndt7_union as ndt.ndt7, and ask that you update your queries accordingly.  Although we are not aware of any problems with the data from the dynamic fleet, we have not yet systematically validated the calibration of this data.&lt;/p&gt;

&lt;p&gt;Moreover, if you’re currently using the unified or intermediate views and are unable to easily switch to ndt7 or ndt7_union, we’d like to hear from you. These views were originally developed to support the transition from ndt5 to ndt7, but as part of an upcoming update, we plan to revise our recommendations regarding their use (today, ndt5 accounts for less than 0.5% of newly collected data).&lt;/p&gt;

&lt;p&gt;If you have any questions about these changes or how they might affect your work, please let us know at &lt;a href=&quot;mailto:support@measurementlab.net&quot;&gt;support@measurementlab.net&lt;/a&gt;&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:1&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;The term “dynamic” refers to the more dynamic way of setup and operation, as well as the availability of the servers. &lt;a href=&quot;#fnref:1&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;</content><author><name>Matt Mathis, Roberto D’Auria, Nathan Kinkade, Pavlos Sermpezis, Melissa Newcomb</name></author><category term="data" /><category term="bigquery" /><category term="Ndt7" /><category term="announcement" /><summary type="html">A new dataset of ndt7 measurements is now available in BigQuery, offering access to measurements from a set of servers whose data had not been previously published.</summary></entry><entry><title type="html">OMG! Summary of the 3rd Open Measurement Gathering (OMG) Ask Me Anything (AMA) event</title><link href="/blog/omg3/" rel="alternate" type="text/html" title="OMG! Summary of the 3rd Open Measurement Gathering (OMG) Ask Me Anything (AMA) event" /><published>2025-07-15T00:00:00+00:00</published><updated>2025-07-15T00:00:00+00:00</updated><id>/blog/omg3</id><content type="html" xml:base="/blog/omg3/">&lt;p&gt;On June 25, 2025, the Open Measurement Gathering (OMG) held a public event, “&lt;em&gt;Open Measurement Gathering AMA”,&lt;/em&gt; featuring &lt;a href=&quot;https://censoredplanet.org/&quot;&gt;Censored Planet&lt;/a&gt;, &lt;a href=&quot;https://ioda.live&quot;&gt;IODA&lt;/a&gt;, &lt;a href=&quot;https://ooni.org/&quot;&gt;OONI&lt;/a&gt;, and &lt;a href=&quot;https://www.measurementlab.net/&quot;&gt;Measurement Lab&lt;/a&gt;. &lt;!--more--&gt;
This was a chance for the OMG group to share project updates, future plans, and gather questions and feedback from the Internet freedom community. Each measurement group presented for 30 minutes followed by Q&amp;amp;A.&lt;/p&gt;

&lt;p&gt;This public virtual event was inspired by the past two OMG convenings (public &lt;a href=&quot;https://www.measurementlab.net/blog/open-measurement-gathering-1-public-report/&quot;&gt;reports 1&lt;/a&gt; and &lt;a href=&quot;https://www.measurementlab.net/blog/open-measurement-gathering-2/&quot;&gt;2&lt;/a&gt;) at which OMG groups have shared exciting updates to their platforms, tools, and/or datasets before sharing with the broader Internet freedom community. For the third OMG convening, the groups decided to share previewing updates publicly to encourage community feedback.&lt;/p&gt;

&lt;p&gt;During the event, OMG groups gathered feedback and answered questions from the Internet freedom community directly.&lt;/p&gt;

&lt;p&gt;Detailed summaries of the presentations and Key Community Questions can be found in the OMG3 public &lt;a href=&quot;/publications/OMG3_report_July2025.pdf&quot;&gt;report&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;You can also watch the presentations and listen to the full Q&amp;amp;A for each session. Links to each group’s presentation:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;a href=&quot;https://www.youtube.com/playlist?list=PLEszjns3sXFGsa42CYPxrQrFPAfA7v71v&quot;&gt;&lt;strong&gt;Full OMG AMA Playlist&lt;/strong&gt;&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Censored Planet&lt;/strong&gt; &lt;a href=&quot;https://docs.google.com/presentation/d/18DOCyU4yRpxMhdsGDoDe9udGuiMgxxQeHZs8C5vo600/edit?usp=drive_link&quot;&gt;Slides&lt;/a&gt; and &lt;a href=&quot;https://youtu.be/5MqsYkBDXYk?feature=shared&quot;&gt;Recording&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;Measurement Lab&lt;/strong&gt; &lt;a href=&quot;https://docs.google.com/presentation/d/1U_BnDTpAnXsNS4DMObi3aiGDKVWZH__xc7KSMqlli6A/edit?usp=sharing&quot;&gt;Slides&lt;/a&gt; and &lt;a href=&quot;https://youtu.be/tBBpW497kME?feature=shared&quot;&gt;Recording&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;OONI&lt;/strong&gt; &lt;a href=&quot;https://drive.google.com/file/d/1Xnyga1QNQQis_INsAK2Ky2puMuZBrreL/view?usp=drive_link&quot;&gt;Slides&lt;/a&gt; and &lt;a href=&quot;https://youtu.be/WuPu3T2Vrqk?feature=shared&quot;&gt;Recording&lt;/a&gt;&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;IODA&lt;/strong&gt; &lt;a href=&quot;https://docs.google.com/presentation/d/1Q5ia6f4a3mzcprfpiCyd8Gmgthu0qrtd/edit?usp=sharing&amp;amp;ouid=110513362222638557784&amp;amp;rtpof=true&amp;amp;sd=true&quot;&gt;Slides&lt;/a&gt;  and &lt;a href=&quot;https://youtu.be/BSV9gktaDjs?feature=shared&quot;&gt;Recording&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The OMG AMA event was especially intended  for advocacy organizations, digital rights researchers, anti-censorship tool developers, journalists, lawyers, activists, policy makers, and funders. At its peak, we had about 60 folks joining us from around the world. The OMG groups truly appreciate those who could attend, and we hope to host more joint events in the future!&lt;/p&gt;</content><author><name>Pavlos Sermpezis, Melissa Newcomb</name></author><category term="event" /><category term="censorship measurement" /><category term="community" /><summary type="html">On June 25, 2025, the Open Measurement Gathering (OMG) held a public event, “Open Measurement Gathering AMA”, featuring Censored Planet, IODA, OONI, and Measurement Lab.</summary></entry><entry><title type="html">Measurement Lab publishes the Internet Quality Barometer Framework</title><link href="/blog/iqb/" rel="alternate" type="text/html" title="Measurement Lab publishes the Internet Quality Barometer Framework" /><published>2025-06-30T00:00:00+00:00</published><updated>2025-06-30T00:00:00+00:00</updated><id>/blog/iqb</id><content type="html" xml:base="/blog/iqb/">&lt;p&gt;Measurement Lab is happy to announce the publication of the Internet Quality Barometer Framework! &lt;!--more--&gt;
The Internet Quality Barometer (IQB) is an initiative funded by the Internet Society Foundation’s Research Grant program motivated by the need to redefine how we measure and understand Internet performance to keep pace with evolving technological demands and user expectations.&lt;/p&gt;

&lt;div style=&quot;background-color: #E7F1FF; padding: 10px;&quot;&gt;
	&lt;p&gt;&lt;strong&gt;Read more about the IQB framework in our&lt;/strong&gt;&lt;/p&gt;
	&lt;ul&gt;
	  &lt;li&gt;&lt;a href=&quot;/publications/IQB_report_2025.pdf&quot; target=&quot;_blank&quot; rel=&quot;noopener noreferrer&quot;&gt;Detailed report&lt;/a&gt;&lt;/li&gt;
	  &lt;li&gt;&lt;a href=&quot;/publications/IQB_executive_summary_2025.pdf&quot; target=&quot;_blank&quot; rel=&quot;noopener noreferrer&quot;&gt;Executive summary&lt;/a&gt;&lt;/li&gt;
	  &lt;li&gt;ACM IMC 2025 &lt;a href=&quot;https://arxiv.org/pdf/2509.19034&quot; target=&quot;_blank&quot; rel=&quot;noopener noreferrer&quot;&gt;poster paper&lt;/a&gt;&lt;/li&gt;
	&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;p&gt;For decades, policymakers, advocates, Internet researchers, Internet service providers and more have relied on speed test data to tell us if our Internet connections are performing well. But as the Internet has evolved so have its use cases and “high-speed” alone is no longer representative of “high-quality” Internet.&lt;/p&gt;

&lt;!-- ![Image](/images/blog/2025-06-30_iqb/IQB Score.png) --&gt;
&lt;p&gt;&lt;img src=&quot;/images/blog/2025-06-30_iqb/IQB Score.png&quot; alt=&quot;IQB score&quot; style=&quot;width: 40%; height: auto;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;IQB is a comprehensive framework for collecting data and calculating a composite score, the “&lt;em&gt;IQB Score&lt;/em&gt;”, which reflects the quality of Internet experience. IQB takes a more holistic approach than “speed tests” and evaluates Internet performance across various use cases (web browsing, video streaming, online gaming, etc.), each with its own specific network requirements (latency, throughput, etc.).&lt;/p&gt;

&lt;!-- ![Image](/images/blog/2025-06-30_iqb/Audio Streaming.png) --&gt;
&lt;p&gt;&lt;img src=&quot;/images/blog/2025-06-30_iqb/Audio Streaming.png&quot; alt=&quot;IQB audio streaming use case&quot; style=&quot;width: 60%; height: auto;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;Plenty of Internet quality data, tools, and methodologies exist, but it is unclear how to effectively integrate them with one another.&lt;/p&gt;

&lt;!-- ![Image](/images/blog/2025-06-30_iqb/Policymaker.png) --&gt;
&lt;p&gt;&lt;img src=&quot;/images/blog/2025-06-30_iqb/Policymaker.png&quot; alt=&quot;IQB policymaker map&quot; style=&quot;width: 60%; height: auto;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;IQB’s primary goal is to fill that gap and build consensus for a more nuanced definition of Internet quality. By moving beyond a narrow focus on speed, IQB seeks to provide stakeholders with actionable insights that support smarter policies and a more equitable Internet.&lt;/p&gt;

&lt;p&gt;To create the IQB framework, M-Lab engaged with more than 60 experts across various fields, including academic network research, public policy, digital inclusion advocacy, Internet service provision, speed test data analysis, content provision, and other related domains from November 2023 to March 2025.&lt;/p&gt;

&lt;!-- ![Image](/images/blog/2025-06-30_iqb/IQB framework.png) --&gt;
&lt;p&gt;&lt;img src=&quot;/images/blog/2025-06-30_iqb/IQB framework.png&quot; alt=&quot;IQB framework&quot; style=&quot;width: 60%; height: auto;&quot; /&gt;&lt;/p&gt;

&lt;p&gt;This is the first phase of the project and we’ll continue iterating on. In future phases, we aim to build an accessible tool for putting the framework to use. The first public report introduces the IQB framework that will be the basis for an IQB tool that will provide stakeholders with actionable insights to support smarter policies and a more equitable Internet.&lt;/p&gt;

&lt;p&gt;For more details about the IQB Framework, read the detailed &lt;a href=&quot;/publications/IQB_report_2025.pdf&quot;&gt;report&lt;/a&gt; or the &lt;a href=&quot;/publications/IQB_executive_summary_2025.pdf&quot;&gt;executive summary&lt;/a&gt;, and our ACM IMC 2025 &lt;a href=&quot;https://arxiv.org/pdf/2509.19034&quot;&gt;poster paper&lt;/a&gt;!&lt;/p&gt;

&lt;!-- ![Image](/images/blog/2025-06-30_iqb/IQB score weights.png) --&gt;
&lt;p&gt;&lt;img src=&quot;/images/blog/2025-06-30_iqb/IQB score weights.png&quot; alt=&quot;IQB score weights&quot; style=&quot;width: 40%; height: auto;&quot; /&gt;&lt;/p&gt;</content><author><name>Pavlos Sermpezis, Melissa Newcomb, Lai Yi Ohlsen</name></author><category term="publication" /><category term="announcement" /><summary type="html">Measurement Lab is happy to announce the publication of the Internet Quality Barometer Framework!</summary></entry><entry><title type="html">IP Route Survey (IPRS) data published in M-Lab</title><link href="/blog/iprs-data-on-mlab/" rel="alternate" type="text/html" title="IP Route Survey (IPRS) data published in M-Lab" /><published>2025-06-16T00:00:00+00:00</published><updated>2025-06-16T00:00:00+00:00</updated><id>/blog/iprs-data-on-mlab</id><content type="html" xml:base="/blog/iprs-data-on-mlab/">&lt;p&gt;We are excited to announce that the Dioptra research group at Sorbonne University is making its &lt;a href=&quot;https://www.measurementlab.net/tests/iprs/&quot;&gt;IP Route Survey (IPRS) available on M-Lab&lt;/a&gt;. &lt;!--more--&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://iprs.dioptra.io/&quot;&gt;IPRS&lt;/a&gt; is an initiative to continuously monitor IP-level routing across the internet. This is done through the regular collection of traceroute-style measurements from multiple vantage points towards a significant portion of the internet’s routable address blocks. IPRS consists of distributed route traces from, currently, 10 vantage points to all routable IPv4 prefixes.The survey is conducted by the &lt;a href=&quot;https://dioptra.io/&quot;&gt;Dioptra&lt;/a&gt; research group at &lt;a href=&quot;https://www.sorbonne-universite.fr/en/&quot;&gt;Sorbonne&lt;/a&gt; University’s &lt;a href=&quot;https://www.lip6.fr/?LANG=en&quot;&gt;LIP6&lt;/a&gt; computer science laboratory.&lt;/p&gt;

&lt;p&gt;IPRS is similar to CAIDA’s Archipelago (&lt;a href=&quot;https://www.caida.org/projects/ark/&quot;&gt;Ark&lt;/a&gt;) data, consisting of multipath route traces. The data is available in the &lt;strong&gt;iprs1&lt;/strong&gt; schema, designed to be consistent with &lt;strong&gt;scamper1&lt;/strong&gt; schema used for M-Lab’s existing large collection of traceroutes. We hope that the compatible formats will make it easier for researchers to use both datasets.&lt;/p&gt;

&lt;p&gt;With the IPRS data, you can get a broader picture of routes through the internet at times when particular M-Lab tests were being performed. This could be useful if you are examining anomalies, for instance, to better understand if those anomalies were restricted to the path over which tests were run, or were part of a broader phenomenon.&lt;/p&gt;

&lt;p&gt;IPRS data is gathered using the &lt;a href=&quot;https://iris.dioptra.io/#/&quot;&gt;Iris&lt;/a&gt; infrastructure, the &lt;a href=&quot;https://github.com/dioptra-io/zeph&quot;&gt;Zeph&lt;/a&gt; and &lt;a href=&quot;https://github.com/dioptra-io/diamond-miner&quot;&gt;Diamond-Miner&lt;/a&gt; algorithms, and the &lt;a href=&quot;https://github.com/dioptra-io/caracal&quot;&gt;Caracal&lt;/a&gt; prober as described in &lt;a href=&quot;https://dl.acm.org/doi/10.1145/3523230.3523232&quot;&gt;&lt;em&gt;Zeph &amp;amp; Iris map the internet: A resilient reinforcement learning approach to distributed IP route tracing&lt;/em&gt;&lt;/a&gt;, an ACM SIGCOMM Computer Communication Review article from 2022. This work was funded by the French Ministry of Armed Forces on a cybersecurity grant.&lt;/p&gt;

&lt;p&gt;For purposes of economizing storage space and processing on M-Lab’s BigQuery, we provide a subset of the data. The entire set and historical data from early 2021, IPv6 data, and other datasets, are available upon request from Dioptra.  Specifically, Dioptra has provided the following two tables:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;iprs_data1&lt;/strong&gt; IPRS data (a representative subset of all IPRS data)&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;iprs_index1&lt;/strong&gt; an index of all IPRS meatada data&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can access IPRS data and metadata in the BigQuery tables &lt;strong&gt;iprs_data1&lt;/strong&gt; and &lt;strong&gt;iprs_index1&lt;/strong&gt; in the &lt;strong&gt;sorbonne&lt;/strong&gt; dataset.  For details on how IPRS data was converted from Iris native format to be compatible with &lt;strong&gt;scamper1&lt;/strong&gt;, please see the &lt;a href=&quot;https://www.measurementlab.net/data/&quot;&gt;Data&lt;/a&gt; page.&lt;/p&gt;

&lt;p&gt;We encourage you to test this new dataset and write to us at &lt;a href=&quot;mailto:iprs@dioptra.io&quot;&gt;iprs@dioptra.io&lt;/a&gt; with your feedback!&lt;/p&gt;</content><author><name>Timur Friedman, Saied Kazemi, Elena Nardi</name></author><category term="data" /><category term="partnership" /><category term="bigquery" /><category term="traceroute" /><category term="announcement" /><summary type="html">We are excited to announce that the Dioptra research group at Sorbonne University is making its IP Route Survey (IPRS) available on M-Lab.</summary></entry><entry><title type="html">How M-Lab Determines User Location and Selects Servers</title><link href="/blog/improving-m-lab-geolocation/" rel="alternate" type="text/html" title="How M-Lab Determines User Location and Selects Servers" /><published>2025-05-27T00:00:00+00:00</published><updated>2025-05-27T00:00:00+00:00</updated><id>/blog/improving-m-lab-geolocation</id><content type="html" xml:base="/blog/improving-m-lab-geolocation/">&lt;p&gt;This blog post describes the two geolocation systems M-Lab uses for server selection and data annotation and how researchers can leverage metadata about the server selection to identify potentially erroneous client geolocation results in the annotated data. &lt;!--more--&gt;&lt;/p&gt;

&lt;div style=&quot;background-color: #E7F1FF; padding: 10px;&quot;&gt;
  This post was contributed by a member of the M-Lab community. 
  We value input from our community and invite others to share their insights on internet measurement, research, and related topics. 
  &lt;p&gt;Interested in contributing? Please email us at &lt;strong&gt;hello@measurementlab.net&lt;/strong&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;&lt;br /&gt;&lt;/p&gt;

&lt;h2 id=&quot;geolocation-at-test-time&quot;&gt;Geolocation at Test Time&lt;/h2&gt;

&lt;p&gt;The user’s location for server selection is determined using &lt;strong&gt;metadata from the HTTP request&lt;/strong&gt;, including the IP address and other client hints provided by &lt;strong&gt;Google’s App Engine&lt;/strong&gt;. This method offers a relatively accurate view of where the user is, and it’s used to make sure the test runs against an appropriate nearby server.&lt;/p&gt;

&lt;h2 id=&quot;geolocation-in-the-public-dataset&quot;&gt;Geolocation in the Public Dataset&lt;/h2&gt;

&lt;p&gt;When test results are stored in BigQuery, M-Lab uses a different system to determine location: &lt;strong&gt;MaxMind’s GeoLite2 database&lt;/strong&gt;. The processing pipeline applies this annotation within a few hours of each test using a fresh copy of the MaxMind database. This database maps IP addresses to cities and countries, but it may be less accurate in certain contexts, especially for mobile users, VPNs, or users behind NATs. Specifically, &lt;a href=&quot;https://support.maxmind.com/hc/en-us/articles/4407630607131-Geolocation-Accuracy&quot;&gt;MaxMind reports accuracies of 99.8% and 66% at the country and city levels,&lt;/a&gt; respectively.&lt;/p&gt;

&lt;p&gt;Depending on the type of research being done with the data, country-level accuracy may be sufficient, but other applications benefit from identifying erroneous geolocation results at finer granularities. Below, we describe how to leverage these two geolocation systems to identify potentially incorrect client geolocation results in the BigQuery data.&lt;/p&gt;

&lt;h2 id=&quot;identifying-geolocation-inconsistencies&quot;&gt;Identifying Geolocation Inconsistencies&lt;/h2&gt;

&lt;p&gt;Recall, there are two geolocation systems in play:&lt;/p&gt;

&lt;ul&gt;
  &lt;li&gt;&lt;strong&gt;Google/App Engine geolocation&lt;/strong&gt;: Used at test time to select a server.&lt;/li&gt;
  &lt;li&gt;&lt;strong&gt;MaxMind geolocation&lt;/strong&gt;: Used in the dataset to report user location.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This can lead to inconsistencies between where the test was &lt;em&gt;served&lt;/em&gt; and where the user &lt;em&gt;appears&lt;/em&gt; to be in the data that we can leverage to identify potentially incorrect client geolocation in the BigQuery data.&lt;/p&gt;

&lt;p&gt;Specifically, we recommend looking at cases where the “metro_rank”&lt;sup id=&quot;fnref:1&quot; role=&quot;doc-noteref&quot;&gt;&lt;a href=&quot;#fn:1&quot; class=&quot;footnote&quot; rel=&quot;footnote&quot;&gt;1&lt;/a&gt;&lt;/sup&gt; field is 0 (indicating that the &lt;strong&gt;Locate API&lt;/strong&gt; served it to the closest site). You can then compute the nearest server metro to the client location published in BigQuery. If the nearest server metro does not match the metro of the server that was used for the test, this indicates a potential discrepancy between the geolocation result of the Locate API (used for server selection) and the one of MaxMind (used to annotate the data). Note that this does not tell you which geolocation result was &lt;em&gt;correct,&lt;/em&gt; but it indicates a potential error in the client geolocation that can be filtered for applications that are sensitive to the specific client location.&lt;/p&gt;

&lt;p&gt;The SQL below gives an example of performing this process.&lt;/p&gt;

&lt;h2 id=&quot;query-to-select-consistent-geolocations&quot;&gt;Query to Select Consistent Geolocations&lt;/h2&gt;

&lt;p&gt;The following query identifies and keeps only the client IPs that show consistent geolocation behavior across server selection and reporting.&lt;/p&gt;

&lt;div class=&quot;language-plaintext highlighter-rouge&quot;&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre class=&quot;highlight&quot;&gt;&lt;code&gt;-- Replace ${DAY} with your date of choice in &apos;YYYY-MM-DD&apos; format, e.g., &apos;2025-05-01&apos;
-- This query filters for client IPs that are located close to their nearest M-Lab server
-- and are likely not affected by test location inconsistencies.

WITH All_Client_Locations AS (
  -- Step 1: Get all unique client locations (by city + state + country),
  -- along with their lat/lon on the given day
  SELECT DISTINCT
  CONCAT(client.Geo.City, &apos;-&apos;, client.Geo.Subdivision1ISOCode, &apos;-&apos;, client.Geo.CountryCode) AS client_city,
  client.Geo.Latitude  AS client_lat,
  client.Geo.Longitude AS client_lon
  FROM `measurement-lab.ndt.ndt7`
  WHERE
  date = &apos;${DAY}&apos; -- 🟡 Replace this with your desired date
  AND client.Geo.Latitude IS NOT NULL
  AND client.Geo.Longitude IS NOT NULL
),

All_Server_Locations AS (
  -- Step 2: Get all unique server locations with valid lat/lon and city
  SELECT DISTINCT
  server.Geo.City     AS server_city,
  server.Geo.Latitude   AS server_lat,
  server.Geo.Longitude  AS server_lon
  FROM `measurement-lab.ndt.ndt7`
  WHERE
  date = &apos;${DAY}&apos; -- 🟡 Replace this with your desired date
  AND server.Geo.Latitude IS NOT NULL
  AND server.Geo.Longitude IS NOT NULL
  AND server.Geo.City IS NOT NULL
),

MinDistancesPerCity AS (
  -- Step 3: For each client city, compute the great-circle distance to its closest server
  SELECT
  c.client_city,
  MIN(ST_DISTANCE(
      ST_GEOGPOINT(c.client_lon, c.client_lat),
      ST_GEOGPOINT(s.server_lon, s.server_lat)
  ) / 1000) AS min_gcd_km -- convert meters to kilometers
  FROM All_Client_Locations c
  CROSS JOIN All_Server_Locations s
  GROUP BY c.client_city
),

DistanceCalc AS (
  -- Step 4: For each test, calculate:
  -- - the distance from client to server
  -- - the metro rank
  -- - the client type (e.g., ist or not)
  SELECT
  CONCAT(client.Geo.City, &apos;-&apos;, client.Geo.Subdivision1ISOCode, &apos;-&apos;, client.Geo.CountryCode) AS client_city,
  raw.ClientIP AS client_ip,
  ST_DISTANCE(
    ST_GEOGPOINT(client.Geo.Longitude, client.Geo.Latitude),
    ST_GEOGPOINT(server.Geo.Longitude, server.Geo.Latitude)
  ) / 1000 AS gcd_km,
  cm.Value AS metro_rank,
  (
    SELECT cm2.Value
    FROM UNNEST(ndt.raw.Download.ClientMetadata) AS cm2
    WHERE cm2.Name = &apos;client_name&apos;
  ) AS client_type
  FROM `measurement-lab.ndt.ndt7` ndt
  CROSS JOIN UNNEST(ndt.raw.Download.ClientMetadata) cm
  WHERE
  date = &apos;${DAY}&apos; -- 🟡 Replace this with your desired date
  AND cm.Name = &apos;metro_rank&apos;
),

FilteredIPs AS (
  -- Step 5: Keep tests where:
  -- - the distance from client to chosen server is within 250km of the *nearest* server
  -- - OR the client is not of type &apos;ist&apos; (e.g., from a automated tool)
  SELECT
  d.client_ip,
  md.min_gcd_km
  FROM DistanceCalc d
  JOIN MinDistancesPerCity md
  ON d.client_city = md.client_city
  WHERE
  (
    ABS(d.gcd_km - md.min_gcd_km) &amp;lt; 250
    AND d.metro_rank IN (&apos;0&apos;, &apos;1&apos;)
  )
  OR (d.client_type != &apos;ist&apos;)
),

ConsistentIPs AS (
  -- Step 6: Return only the unique set of IPs that passed the filter
  SELECT DISTINCT client_ip
  FROM FilteredIPs
)

-- Final result: set of IPs that have geolocation + test location consistency
SELECT
  *
FROM ConsistentIPs
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;

&lt;h2 id=&quot;impact-of-filtering-on-dataset-size&quot;&gt;Impact of Filtering on Dataset Size&lt;/h2&gt;

&lt;p&gt;The effectiveness of our filtering depends on how many servers are available nearby. Figure 1 defines the region in which each server is the nearest server. In dense regions, cells are small and the filter can finely distinguish between client locations; in sparse regions (e.g., much of Africa), large cells group many cities under the same server, making the filter less selective.&lt;/p&gt;

&lt;!-- ![Image](/images/blog/2025-05-improving-m-lab-geolocation/voronoi-partition-m-lab-servers.png) --&gt;
&lt;p&gt;&lt;img src=&quot;/images/blog/2025-05-improving-m-lab-geolocation/voronoi-partition-m-lab-servers.png&quot; alt=&quot;Voronoi M-lab servers&quot; style=&quot;width: 100%; height: auto;&quot; /&gt;
&lt;strong&gt;Figure 1.&lt;/strong&gt; Voronoi partition of the globe overlaid on M-Lab server locations. Each colored cell shows the region of longitude/latitude space closest to its red-dot server by great-circle distance.&lt;/p&gt;

&lt;p&gt;To assess the impact of filtering, we computed the number of unique client IPs retained versus removed when applying the consistency criteria on the 1st of April 2025. The filter excludes 18% of the measured IPs in Africa to 7.88% in South America. These results highlight a trade-off: while filtering improves geolocation reliability, it also reduces coverage, potentially in regions where measurement density is already limited. Therefore, users should really run these queries only if their results hinge on accurate geolocations.&lt;/p&gt;

&lt;table&gt;
  &lt;thead&gt;
    &lt;tr&gt;
      &lt;th style=&quot;text-align: center&quot;&gt;Continent&lt;/th&gt;
      &lt;th style=&quot;text-align: center&quot;&gt;Total IPs&lt;/th&gt;
      &lt;th style=&quot;text-align: center&quot;&gt;Consistent IPs&lt;/th&gt;
      &lt;th style=&quot;text-align: center&quot;&gt;Removed IPs&lt;/th&gt;
      &lt;th style=&quot;text-align: center&quot;&gt;% Removed&lt;/th&gt;
    &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
    &lt;tr&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;Africa&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;112,293&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;91,605&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;20,688&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;18.42%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;Asia&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;1,123,951&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;933,007&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;190,944&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;16.99%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;Europe&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;526,951&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;463,247&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;63,704&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;12.09%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;Oceania&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;58,656&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;52,316&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;6,340&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;10.81%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;North America&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;519,671&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;465,723&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;53,948&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;10.38%&lt;/td&gt;
    &lt;/tr&gt;
    &lt;tr&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;South America&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;127,903&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;117,822&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;10,081&lt;/td&gt;
      &lt;td style=&quot;text-align: center&quot;&gt;7.88%&lt;/td&gt;
    &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;

&lt;h2 id=&quot;about-the-authors&quot;&gt;About the authors&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Loqman Salamatian&lt;/strong&gt; is a Ph.D. candidate in Computer Science at Columbia University, advised by Professors Ethan Katz-Bassett, Vishal Misra, and Daniel Rubenstein. His research focuses on Internet measurement, topology inference, and the relationship between virtual and geographical aspects of network structure. While grounded in theory and mathematical modeling, his work emphasizes building systems that leverage these foundations to uncover hidden properties and dynamics of the Internet.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Phillipa Gill&lt;/strong&gt; is a research scientist at Google. Her research interests include network measurement, BGP security, broadband connectivity and network operations. She was previously a faculty member at the University of Massachusetts – Amherst and Stony Brook University.&lt;/p&gt;

&lt;div class=&quot;footnotes&quot; role=&quot;doc-endnotes&quot;&gt;
  &lt;ol&gt;
    &lt;li id=&quot;fn:1&quot; role=&quot;doc-endnote&quot;&gt;
      &lt;p&gt;The metro_rank flag can be found in the ndt.raw.Download.ClientMetadata array &lt;a href=&quot;#fnref:1&quot; class=&quot;reversefootnote&quot; role=&quot;doc-backlink&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
    &lt;/li&gt;
  &lt;/ol&gt;
&lt;/div&gt;</content><author><name>Loqman Salamatian, Phillipa Gill</name></author><category term="data" /><category term="accuracy" /><category term="research" /><category term="community" /><summary type="html">This blog post describes the two geolocation systems M-Lab uses for server selection and data annotation and how researchers can leverage metadata about the server selection to identify potentially erroneous client geolocation results in the annotated data.</summary></entry></feed>