To make our traceroute data in BigQuery more useful, researchers have sought an easy way to reconstruct the path of hops for the same test. This task was particularly hard because the schema, which was designed many years ago, put the hops of the same test in different rows.
To address this need from many of our partners and researchers, M-Lab is delighted to announce that the
traceroute BigQuery table in the
aggregate dataset is now available to the public. The new
traceroute schema has one test per row, and all hops for a single test are inside the same row.
- Stock Linux 4.19 LTS kernels with modern TCP and Cubic congestion control
- Standard instrumentation for all experiments using tcp-info
- Virtualization and container management using Kubernetes and Docker
- Reimplementation of the NDT server
Earlier this month, M-Lab published updates to our policies after completing a comprehensive review to ensure our compliance with the EU General Data Protection Regulation (GDPR) and in preparation for the M-Lab 2.0 platform modernization update that will be rolled out this fall. This post outlines the changes and additions to our policies for the general public, for experiment developers hosting tests on the M-Lab platform, and for partners who provide hosting for M-Lab servers.
M-Lab is working on replacing the current traceroute BigQuery table with new schema, which will put all hops of one test in one row of BigQuery table. The new table will have all the information in the current table but make the search of hops within one test much easier. To make this happen, we will stop the new data feed of current traceroute BigQuery table in early July, 2019. The details of new schema will be published once the conversion of all data to BigQuery tables with the new traceroute schema is completed and available to the public.
Earlier this year, M-Lab published blog post outlining our new ETL pipeline and transition to new BigQuery tables. That post also outlined where we’ve saved our datasets, tables, and views in BigQuery historically, and recommended tables and views for most researchers to use. At that time we also implemented semantic versioning to new dataset and table releases at that time, and began publishing BigQuery views that unify our NDT data across multiple schema iterations and migrations.