Naming BQ Datasets after M-Lab Measurement Services & Data Types

Earlier this year, M-Lab published blog post outlining our new ETL pipeline and transition to new BigQuery tables. That post also outlined where we’ve saved our datasets, tables, and views in BigQuery historically, and recommended tables and views for most researchers to use. At that time we also implemented semantic versioning to new dataset and table releases at that time, and began publishing BigQuery views that unify our NDT data across multiple schema iterations and migrations.

Read More

AFRICOMM 2018 and AFRINIC-29

M-Lab and researchers at AFRICOMM-2018 AFRICOMM 2018. Left to right: Amreesh Phokeer (AFRINIC), Josiah Chavula (University of Capetown), Georgia Bullen (M-Lab), Antoine Delvaux (perfSonar), Stephen Soltesz (M-Lab).

In late November 2018, M-Lab was invited to the Internet Measurement Workshop at AFRINIC-29 in Tunisia and to give a keynote about M-Lab and open internet measurement at AFRICOMM 2018 in Senegal. Both trips were a fantastic opportunity to deepen our relationship with researchers focused on the African Internet, learn more about how our platform is serving community needs, foster conversation around open Internet measurement, and identify opportunities for further collaboration, research and tool development to better support the Internet measurement, research and policy community in Africa.

Read More

SIGCOMM 2018 Hackathon Outcomes

SIGCOMM Hackathon - M-Lab Participants Working

M-Lab had the pleasure of attending the first ever SIGCOMM hackathon on August 25, 2018, at the Nokia Skypark headquarters in Budapest, Hungary. The hackathon, sponsored by Nokia, DECIX, and Netflix, invited network research faculty, students, and industry professionals from around the world to form teams and develop tools, new features or analyses during the Saturday following the SIGCOMM conference.

Read More

M-Lab 10 Year Anniversary Event

We’ve reached a point in human history where, for many of us, the Internet has become a standard presence in our daily lives. In the United States, the Internet is simply part of how many of us engage with the world. In other countries (and parts of this one), the Internet remains unaffordable, unreliable, and inaccessible. The Internet unites us in many ways, and at the center of work on the future of the Internet is a dedicated community of experts exploring the questions that will move the Internet to the next level of its evolution: What is an open Internet? What is a healthy Internet? What factors contribute to the Internet ecosystem’s health?

Read More

Switch discard investigation for early 2018

On February 1st, 2018, during a regular data quality review, we identified an increase in switch discards at sites with 10Gbps equipment connected to 1Gbps uplinks. We used our switch telemetry data to assess whether there were any negative consequences for tests contained in our SideStream or NDT data sets, and then we used the same data sets to determine whether our remediation strategy had any negative effects. In both cases, we found no observable effects, indicating that everything was below the noise floor for Internet performance data.

Read More

M-Lab is Turning 10!

M-Lab 10 Year Anniversary Graphic

  • When: Aug. 7, 2018 - Aug. 8, 2018, 9AM - 5PM
  • Where: New America, 740 15th St NW #900, Washington, D.C. 20005

Measurement Lab is turning 10! On August 7 and 8, we look forward to gathering the Measurement Lab community to showcase how the platform has evolved, learn from you about how you are using M-Lab, and discuss how we plan for the next 10 years of measuring the Internet and providing public data to the world. So much has changed over the last 10 years (and that’s not just our expanding volume of longitudinal data!), come celebrate, brainstorm, analyze, and share with us.

Read More

New ETL Pipeline and Transition to New BigQuery Tables

Since May 2017, the M-Lab team has been working on an updated, open source pipeline, which pulls raw data from our servers, saves it to Google Cloud Storage, and then parses it into our BigQuery tables. The team is particularly excited about this update because it means that the pipeline no longer relies on closed source libraries.

Read More

Transitioning to a New Backend Pipeline and Data Availability

M-Lab data is collected from distributed experiments hosted on servers all over the world, processed in a pipeline, and published for free in both raw and parsed (structured) formats. The back end processing component for this has served us well for many years, but it’s been showing its age recently. As M-Lab collects an increasing amount of data thanks to new partnerships, we have been concerned that it will not be as reliable.

Read More

Making it Easier to Use M-Lab Data

In January, M-Lab launched a beta test of new BigQuery tables for M-Lab data. Today, M-Lab is pleased to announce that the beta test was successful. The new, faster-performing tables will be M-Lab’s new standard BigQuery tables.

Before we move on to specifics, when we say faster performing, we mean a lot faster. As in, certain queries that used to take over 2 hours now complete in 8 seconds. That means that playing with the data just became a lot more fun.

To help users dig in to this data as quickly and seamlessly as possible, M-Lab has consolidated all of its data documentation and updated it to show how to take advantage of the new tables.

Read More

Announcing improved performance for M-Lab BigQuery data

Today, M-Lab is happy to announce the public beta of new M-Lab BigQuery tables. These tables provide substantially improved performance and reduce the difficulty of writing BigQuery SQL.

Read More

Back to Top