Utilization Dataset

Since June 2016, M-Lab has collected high resolution switch telemetry for each M-Lab server and site uplink and published it as the utilization dataset, with one datatype table: switch.

Utilization Data in Raw Format

Utilization data in raw format can be found in Google Cloud Storage: https://console.developers.google.com/storage/browser/archive-measurement-lab/utilization/.

Utilization Data in BigQuery

M-Lab parses all switch utilization data into BigQuery, and makes query access available for free by subscription to a Google Group. Find out more about how to get access on our BigQuery QuickStart page.

BigQuery Tables/Views for Switch Data:

Switch BigQuery Schema

Field name Type Description
partition_date date The day (in UTC) in which the data was loaded.
test_id string ID of the test. It represents the Google Cloud Storage path, M-Lab server, and file name of log that contains the data generated during the test (e.g. 2019/02/09/mlab1.yul02.measurement-lab.org/20190209T23:00:00-to-20190210T00:00:00-switch.json.gz).
task_filename string The raw data file in Google Cloud Storage from which the test row was parsed.
parse_time timestamp Timestamp of when test data was parsed into BigQuery from Google Cloud Storage.
parser_version string A link to the tagged version of the M-Lab ETL parser which processed the test row.
log_time timestamp Never set for the switch data. The sample.timestamp should be used instead for the sample collection time.
sample record A repeated record with the value and timestamp of each 10 second observation. Typically, there will be 360 samples per hour. Due to system maintenance, or machine restarts, some intervals may contain more or less samples.
sample.timestamp timestamp Timestamp of the beginning of the 10 second time bin.
sample.value float Delta value of the metric during this 10 second time bin.
metric string The canonical metric name for samples, e.g. switch.discards.uplink.tx
hostname string The fully qualified domain name of the machine that collected the data, e.g. mlab2.abc01.measurement-lab.org.
experiment string The fully qualified domain name of the switch that produced the data, e.g. s1.abc01.measurement-lab.org.

Source Code

The switch dataset is produced by the M-Lab Collectd monitoring tool, which can be found on Github: https://github.com/m-lab/collectd-mlab.

Back to Top