M-Lab provides query access to our datasets in BigQuery at no charge to interested users. Following the steps below will allow you to use BigQuery to search M-Lab datasets without charge when the
measurement-lab project is selected in your Google Cloud Platform console, or set as your project in the Google Cloud SDK. Queries from projects you create, saving query results to BigQuery tables, etc. will incur costs to you.
Please follow the steps below to configure free query access. If you have questions, please contact us at email@example.com
Subscribe your Google account to the M-Lab Discuss group
To gain access to M-Lab’s open dataset, you are asked to sign up for the M-Lab discuss mailing list. The M-Lab team uses this mailing list to communicate updates to the project including changes to the data’s format and updates to the platform as well information about upcoming events that M-Lab is hosting or participating in.
The mailing list is also a place for discussion for M-Lab users. We invite our community to ask questions, become familiar with each other’s work, and directly communicate with our team. To facilitate constructive conversation, we ask all participants to adhere to community guidelines. If you prefer to communicate with the M-Lab team directly, we are available at firstname.lastname@example.org
Members of this group are give access to use the
measurement-lab project in the Google Cloud Platform console or Google Cloud SDK so that query charges are paid for by M-Lab.
Setup Access to M-Lab Datasets in the Google Cloud Console
Visit the Google Cloud Console BigQuery page for the measurement-lab project. If you’ve never used BigQuery before, you’ll see a welcome message as shown in the image below. Click “Let’s Go!”
Next, you may need to read and agree to the Terms of Service Updates and click “Accept”, as shown in the image below.
After you agree to the terms of service, your Google Cloud Platform should look like the image below. M-Lab datasets, tables, and views can be displayed by clicking the project text,
measurement-lab in the left column under Resources.
You may now query M-Lab datasets from this project at no charge. Note also that you do not need to activate Google’s $300 credit as shown at the top of the image above, unless you wish to do so.
Setup Access to M-Lab Datasets in the Google Cloud SDK
You may prefer to use the Google Cloud SDK to query M-Lab data. The SDK provides a convenient command line interface for interacting with Google Cloud products.
First, subscribe the Google account you wish to use to the M-Lab Discuss group.
Members of this group are whitelisted to use the
measurement-lab project in the Google Cloud Platform console or Google Cloud SDK so that queries charges are paid for by M-Lab.
Next, download and install the Google Cloud SDK
Use the installation and Quick Start instructions for your operating system.
Authenticate the SDK installation using the Gmail account you subscribed to M-Lab Discuss.
Use the command
$ gcloud auth login and follow the instructions to authenticate:
After installation and authentication, your terminal should show something like:
Finally, set your default project to
BigQuery’s command-line tools should now be available in your terminal and you should be able to query datasets, tables, and views in the
If you are new to BigQuery, we suggest that you next consult the following resources:
- BigQuery Examples
- BigQuery Schema
- Google’s BigQuery documentation
- Querying Date Partitioned Tables
- Google’s bq Command-Line Tool Quickstart documentation
Given the resources and expertise, you can develop your own application that uses M-Lab data using your own Google Cloud Platform Project. To learn more about building a custom application, refer to the Google Cloud Platform Console documentation
Using a Service Account
If you want to query M-Lab data using a service account (e.g., @developer.gserviceaccount.com) associated with your GCP application, please email email@example.com so that we can add your account to M-Lab Discuss manually.
Please note that M-Lab does not extensively test applications that use service accounts to query our data to check which operations may or may not incur billing. Because of the means by which M-Lab must whitelist query users, we cannot guarantee that no charges will be incurred by applications using service accounts which query M-Lab datasets, tables, and views.
M-Lab has confirmed that the following test did not incur billing:
- A GCP project was created using a test Gmail account
- A service account was created with the IAM roles in the test GCP project: BigQuery User, BigQuery Job User, BigQuery Data Viewer
- The service account was added to the M-Lab Discuss Google Group
- The Google Cloud SDK was installed on a Linux computer and configured to use the test GCP project, and its service account
- Several queries were made to datasets within the
- Though these queries appeared in the GCP BigQuery query history for the test project, no billing transactions were present for the queries