ndt5 Protocol - NDT (Network Diagnostic Tool)

The ndt5 protocol in ndt-server supports the many NDT clients which use the previous web100 legacy NDT server. As a part of M-Lab’s platform upgrade in 2019, ndt-server replaced the former web100 based ndt4 protocol with the ndt5 protocol, and added the ndt7 protocol for future client use. New clients will be encouraged to use the ndt7 protocol once the platform transition is completed.

NDT data collected before 2019-07-19 used the Web100 Linux kernel patch for TCP statistics. This data can be found in the NDT web100 (legacy) dataset.

NDT data using the ndt5 protocol collected on or after 2019-07-19 uses tcp-info for all TCP metrics is available in both raw format in Google Cloud Storage and in queryable format in BigQuery.

More details about the ndt5 protocol can be found in the README for ndt5 on Github.

ndt5 BigQuery Schema

Field name Type Description
ParseInfo RECORD Metadata about how the parser processed the measurement.
ParseInfo.TaskFileName STRING GCS URL to the archive containing the test_id for this row.
ParseInfo.ParseTime TIMESTAMP Time that the parser processed this row.
ParseInfo.ParserVersion STRING Version of the parser that processed this row.
test_id STRING Original filename of measurement as written to disk and in the GCS archive.
log_time INTEGER Original measurement collection timestamp.
result RECORD  
result.GitShortCommit STRING GitShortCommit is the Git commit (short form) of the running server code.
result.Version STRING Version is the symbolic version (if any) of the running server code.
result.ServerIP STRING  
result.ServerPort INTEGER  
result.ClientIP STRING  
result.ClientPort INTEGER  
result.StartTime TIMESTAMP  
result.EndTime TIMESTAMP  
result.Control RECORD Metadata for TCP connections to the NDT5 control channel. All NDT5 measurements have a control channel.
result.Control.UUID STRING UUID for control channel.
result.Control.Protocol STRING Protocol used for S2C and C2S measurements. Values include WS, WSS, and PLAIN.
result.Control.MessageProtocol STRING Individual messages are sent with the MessageProtocol. Values include JSON, TLV.
result.Control.ClientMetadata RECORD Client-reported metadata as name/value pairs.
result.Control.ClientMetadata.Name STRING  
result.Control.ClientMetadata.Value STRING  
result.C2S RECORD Metadata for Client-to-Server measurements performed using the NDT5 protocol, e.g. “Upload”.
result.C2S.ServerIP STRING  
result.C2S.ServerPort INTEGER  
result.C2S.ClientIP STRING  
result.C2S.ClientPort INTEGER  
result.C2S.UUID STRING UUID for C2S TCP connection.
result.C2S.StartTime TIMESTAMP  
result.C2S.EndTime TIMESTAMP  
result.C2S.MeanThroughputMbps FLOAT Server calculated average upload rate.
result.C2S.Error STRING  
result.S2C RECORD Metadata for Server-to-Client measurements performed using the NDT5 protocol, e.g. “Download”.
result.S2C.UUID STRING UUID for S2C TCP connection.
result.S2C.ServerIP STRING  
result.S2C.ServerPort INTEGER  
result.S2C.ClientIP STRING  
result.S2C.ClientPort INTEGER  
result.S2C.StartTime TIMESTAMP  
result.S2C.EndTime TIMESTAMP  
result.S2C.MeanThroughputMbps FLOAT Server calculated average download rate.
result.S2C.MinRTT INTEGER The minimum RTT observed during the download measurement.
result.S2C.ClientReportedMbps FLOAT The client reported average download rate.
result.S2C.Error STRING  
result.Upload RECORD  
result.Upload.UUID STRING  
result.Upload.StartTime TIMESTAMP  
result.Upload.EndTime TIMESTAMP  
result.Upload.ServerMeasurements RECORD  
result.Upload.ServerMeasurements.AppInfo RECORD  
result.Upload.ServerMeasurements.AppInfo.NumBytes INTEGER  
result.Upload.ServerMeasurements.AppInfo.ElapsedTime INTEGER  
result.Upload.ServerMeasurements.BBRInfo RECORD  
result.Upload.ServerMeasurements.BBRInfo.ElapsedTime INTEGER  
result.Upload.ServerMeasurements.BBRInfo.MaxBandwidth INTEGER  
result.Upload.ServerMeasurements.BBRInfo.MinRTT INTEGER  
result.Upload.ServerMeasurements.TCPInfo RECORD  
result.Upload.ServerMeasurements.TCPInfo.State INTEGER  
result.Upload.ServerMeasurements.TCPInfo.CAState INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Retransmits INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Probes INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Backoff INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Options INTEGER  
result.Upload.ServerMeasurements.TCPInfo.WScale INTEGER  
result.Upload.ServerMeasurements.TCPInfo.AppLimited INTEGER  
result.Upload.ServerMeasurements.TCPInfo.RTO INTEGER  
result.Upload.ServerMeasurements.TCPInfo.ATO INTEGER  
result.Upload.ServerMeasurements.TCPInfo.SndMSS INTEGER  
result.Upload.ServerMeasurements.TCPInfo.RcvMSS INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Unacked INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Sacked INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Lost INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Retrans INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Fackets INTEGER  
result.Upload.ServerMeasurements.TCPInfo.LastDataSent INTEGER  
result.Upload.ServerMeasurements.TCPInfo.LastAckSent INTEGER  
result.Upload.ServerMeasurements.TCPInfo.LastDataRecv INTEGER  
result.Upload.ServerMeasurements.TCPInfo.LastAckRecv INTEGER  
result.Upload.ServerMeasurements.TCPInfo.PMTU INTEGER  
result.Upload.ServerMeasurements.TCPInfo.RcvSsThresh INTEGER  
result.Upload.ServerMeasurements.TCPInfo.RTT INTEGER  
result.Upload.ServerMeasurements.TCPInfo.RTTVar INTEGER  
result.Upload.ServerMeasurements.TCPInfo.SndSsThresh INTEGER  
result.Upload.ServerMeasurements.TCPInfo.SndCwnd INTEGER  
result.Upload.ServerMeasurements.TCPInfo.AdvMSS INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Reordering INTEGER  
result.Upload.ServerMeasurements.TCPInfo.RcvRTT INTEGER  
result.Upload.ServerMeasurements.TCPInfo.RcvSpace INTEGER  
result.Upload.ServerMeasurements.TCPInfo.TotalRetrans INTEGER  
result.Upload.ServerMeasurements.TCPInfo.PacingRate INTEGER  
result.Upload.ServerMeasurements.TCPInfo.MaxPacingRate INTEGER  
result.Upload.ServerMeasurements.TCPInfo.BytesAcked INTEGER  
result.Upload.ServerMeasurements.TCPInfo.BytesReceived INTEGER  
result.Upload.ServerMeasurements.TCPInfo.SegsOut INTEGER  
result.Upload.ServerMeasurements.TCPInfo.SegsIn INTEGER  
result.Upload.ServerMeasurements.TCPInfo.NotsentBytes INTEGER  
result.Upload.ServerMeasurements.TCPInfo.MinRTT INTEGER  
result.Upload.ServerMeasurements.TCPInfo.DataSegsIn INTEGER  
result.Upload.ServerMeasurements.TCPInfo.DataSegsOut INTEGER  
result.Upload.ServerMeasurements.TCPInfo.DeliveryRate INTEGER  
result.Upload.ServerMeasurements.TCPInfo.BusyTime INTEGER  
result.Upload.ServerMeasurements.TCPInfo.RWndLimited INTEGER  
result.Upload.ServerMeasurements.TCPInfo.SndBufLimited INTEGER  
result.Upload.ServerMeasurements.TCPInfo.Delivered INTEGER  
result.Upload.ServerMeasurements.TCPInfo.DeliveredCE INTEGER  
result.Upload.ServerMeasurements.TCPInfo.BytesSent INTEGER  
result.Upload.ServerMeasurements.TCPInfo.BytesRetrans INTEGER  
result.Upload.ServerMeasurements.TCPInfo.DSackDups INTEGER  
result.Upload.ServerMeasurements.TCPInfo.ReordSeen INTEGER  
result.Upload.ServerMeasurements.TCPInfo.ElapsedTime INTEGER  
result.Upload.ClientMeasurements RECORD  
result.Upload.ClientMeasurements.AppInfo RECORD  
result.Upload.ClientMeasurements.AppInfo.NumBytes INTEGER  
result.Upload.ClientMeasurements.AppInfo.ElapsedTime INTEGER  
result.Upload.ClientMeasurements.BBRInfo RECORD  
result.Upload.ClientMeasurements.BBRInfo.ElapsedTime INTEGER  
result.Upload.ClientMeasurements.BBRInfo.MaxBandwidth INTEGER  
result.Upload.ClientMeasurements.BBRInfo.MinRTT INTEGER  
result.Upload.ClientMeasurements.TCPInfo RECORD  
result.Upload.ClientMeasurements.TCPInfo.State INTEGER  
result.Upload.ClientMeasurements.TCPInfo.CAState INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Retransmits INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Probes INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Backoff INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Options INTEGER  
result.Upload.ClientMeasurements.TCPInfo.WScale INTEGER  
result.Upload.ClientMeasurements.TCPInfo.AppLimited INTEGER  
result.Upload.ClientMeasurements.TCPInfo.RTO INTEGER  
result.Upload.ClientMeasurements.TCPInfo.ATO INTEGER  
result.Upload.ClientMeasurements.TCPInfo.SndMSS INTEGER  
result.Upload.ClientMeasurements.TCPInfo.RcvMSS INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Unacked INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Sacked INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Lost INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Retrans INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Fackets INTEGER  
result.Upload.ClientMeasurements.TCPInfo.LastDataSent INTEGER  
result.Upload.ClientMeasurements.TCPInfo.LastAckSent INTEGER  
result.Upload.ClientMeasurements.TCPInfo.LastDataRecv INTEGER  
result.Upload.ClientMeasurements.TCPInfo.LastAckRecv INTEGER  
result.Upload.ClientMeasurements.TCPInfo.PMTU INTEGER  
result.Upload.ClientMeasurements.TCPInfo.RcvSsThresh INTEGER  
result.Upload.ClientMeasurements.TCPInfo.RTT INTEGER  
result.Upload.ClientMeasurements.TCPInfo.RTTVar INTEGER  
result.Upload.ClientMeasurements.TCPInfo.SndSsThresh INTEGER  
result.Upload.ClientMeasurements.TCPInfo.SndCwnd INTEGER  
result.Upload.ClientMeasurements.TCPInfo.AdvMSS INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Reordering INTEGER  
result.Upload.ClientMeasurements.TCPInfo.RcvRTT INTEGER  
result.Upload.ClientMeasurements.TCPInfo.RcvSpace INTEGER  
result.Upload.ClientMeasurements.TCPInfo.TotalRetrans INTEGER  
result.Upload.ClientMeasurements.TCPInfo.PacingRate INTEGER  
result.Upload.ClientMeasurements.TCPInfo.MaxPacingRate INTEGER  
result.Upload.ClientMeasurements.TCPInfo.BytesAcked INTEGER  
result.Upload.ClientMeasurements.TCPInfo.BytesReceived INTEGER  
result.Upload.ClientMeasurements.TCPInfo.SegsOut INTEGER  
result.Upload.ClientMeasurements.TCPInfo.SegsIn INTEGER  
result.Upload.ClientMeasurements.TCPInfo.NotsentBytes INTEGER  
result.Upload.ClientMeasurements.TCPInfo.MinRTT INTEGER  
result.Upload.ClientMeasurements.TCPInfo.DataSegsIn INTEGER  
result.Upload.ClientMeasurements.TCPInfo.DataSegsOut INTEGER  
result.Upload.ClientMeasurements.TCPInfo.DeliveryRate INTEGER  
result.Upload.ClientMeasurements.TCPInfo.BusyTime INTEGER  
result.Upload.ClientMeasurements.TCPInfo.RWndLimited INTEGER  
result.Upload.ClientMeasurements.TCPInfo.SndBufLimited INTEGER  
result.Upload.ClientMeasurements.TCPInfo.Delivered INTEGER  
result.Upload.ClientMeasurements.TCPInfo.DeliveredCE INTEGER  
result.Upload.ClientMeasurements.TCPInfo.BytesSent INTEGER  
result.Upload.ClientMeasurements.TCPInfo.BytesRetrans INTEGER  
result.Upload.ClientMeasurements.TCPInfo.DSackDups INTEGER  
result.Upload.ClientMeasurements.TCPInfo.ReordSeen INTEGER  
result.Upload.ClientMeasurements.TCPInfo.ElapsedTime INTEGER  
result.Upload.ClientMetadata RECORD  
result.Upload.ClientMetadata.Name STRING  
result.Upload.ClientMetadata.Value STRING  
result.Download RECORD  
result.Download.UUID STRING  
result.Download.StartTime TIMESTAMP  
result.Download.EndTime TIMESTAMP  
result.Download.ServerMeasurements RECORD  
result.Download.ServerMeasurements.AppInfo RECORD  
result.Download.ServerMeasurements.AppInfo.NumBytes INTEGER  
result.Download.ServerMeasurements.AppInfo.ElapsedTime INTEGER  
result.Download.ServerMeasurements.BBRInfo RECORD  
result.Download.ServerMeasurements.BBRInfo.ElapsedTime INTEGER  
result.Download.ServerMeasurements.BBRInfo.MaxBandwidth INTEGER  
result.Download.ServerMeasurements.BBRInfo.MinRTT INTEGER  
result.Download.ServerMeasurements.TCPInfo RECORD  
result.Download.ServerMeasurements.TCPInfo.State INTEGER  
result.Download.ServerMeasurements.TCPInfo.CAState INTEGER  
result.Download.ServerMeasurements.TCPInfo.Retransmits INTEGER  
result.Download.ServerMeasurements.TCPInfo.Probes INTEGER  
result.Download.ServerMeasurements.TCPInfo.Backoff INTEGER  
result.Download.ServerMeasurements.TCPInfo.Options INTEGER  
result.Download.ServerMeasurements.TCPInfo.WScale INTEGER  
result.Download.ServerMeasurements.TCPInfo.AppLimited INTEGER  
result.Download.ServerMeasurements.TCPInfo.RTO INTEGER  
result.Download.ServerMeasurements.TCPInfo.ATO INTEGER  
result.Download.ServerMeasurements.TCPInfo.SndMSS INTEGER  
result.Download.ServerMeasurements.TCPInfo.RcvMSS INTEGER  
result.Download.ServerMeasurements.TCPInfo.Unacked INTEGER  
result.Download.ServerMeasurements.TCPInfo.Sacked INTEGER  
result.Download.ServerMeasurements.TCPInfo.Lost INTEGER  
result.Download.ServerMeasurements.TCPInfo.Retrans INTEGER  
result.Download.ServerMeasurements.TCPInfo.Fackets INTEGER  
result.Download.ServerMeasurements.TCPInfo.LastDataSent INTEGER  
result.Download.ServerMeasurements.TCPInfo.LastAckSent INTEGER  
result.Download.ServerMeasurements.TCPInfo.LastDataRecv INTEGER  
result.Download.ServerMeasurements.TCPInfo.LastAckRecv INTEGER  
result.Download.ServerMeasurements.TCPInfo.PMTU INTEGER  
result.Download.ServerMeasurements.TCPInfo.RcvSsThresh INTEGER  
result.Download.ServerMeasurements.TCPInfo.RTT INTEGER  
result.Download.ServerMeasurements.TCPInfo.RTTVar INTEGER  
result.Download.ServerMeasurements.TCPInfo.SndSsThresh INTEGER  
result.Download.ServerMeasurements.TCPInfo.SndCwnd INTEGER  
result.Download.ServerMeasurements.TCPInfo.AdvMSS INTEGER  
result.Download.ServerMeasurements.TCPInfo.Reordering INTEGER  
result.Download.ServerMeasurements.TCPInfo.RcvRTT INTEGER  
result.Download.ServerMeasurements.TCPInfo.RcvSpace INTEGER  
result.Download.ServerMeasurements.TCPInfo.TotalRetrans INTEGER  
result.Download.ServerMeasurements.TCPInfo.PacingRate INTEGER  
result.Download.ServerMeasurements.TCPInfo.MaxPacingRate INTEGER  
result.Download.ServerMeasurements.TCPInfo.BytesAcked INTEGER  
result.Download.ServerMeasurements.TCPInfo.BytesReceived INTEGER  
result.Download.ServerMeasurements.TCPInfo.SegsOut INTEGER  
result.Download.ServerMeasurements.TCPInfo.SegsIn INTEGER  
result.Download.ServerMeasurements.TCPInfo.NotsentBytes INTEGER  
result.Download.ServerMeasurements.TCPInfo.MinRTT INTEGER  
result.Download.ServerMeasurements.TCPInfo.DataSegsIn INTEGER  
result.Download.ServerMeasurements.TCPInfo.DataSegsOut INTEGER  
result.Download.ServerMeasurements.TCPInfo.DeliveryRate INTEGER  
result.Download.ServerMeasurements.TCPInfo.BusyTime INTEGER  
result.Download.ServerMeasurements.TCPInfo.RWndLimited INTEGER  
result.Download.ServerMeasurements.TCPInfo.SndBufLimited INTEGER  
result.Download.ServerMeasurements.TCPInfo.Delivered INTEGER  
result.Download.ServerMeasurements.TCPInfo.DeliveredCE INTEGER  
result.Download.ServerMeasurements.TCPInfo.BytesSent INTEGER  
result.Download.ServerMeasurements.TCPInfo.BytesRetrans INTEGER  
result.Download.ServerMeasurements.TCPInfo.DSackDups INTEGER  
result.Download.ServerMeasurements.TCPInfo.ReordSeen INTEGER  
result.Download.ServerMeasurements.TCPInfo.ElapsedTime INTEGER  
result.Download.ClientMeasurements RECORD  
result.Download.ClientMeasurements.AppInfo RECORD  
result.Download.ClientMeasurements.AppInfo.NumBytes INTEGER  
result.Download.ClientMeasurements.AppInfo.ElapsedTime INTEGER  
result.Download.ClientMeasurements.BBRInfo RECORD  
result.Download.ClientMeasurements.BBRInfo.ElapsedTime INTEGER  
result.Download.ClientMeasurements.BBRInfo.MaxBandwidth INTEGER  
result.Download.ClientMeasurements.BBRInfo.MinRTT INTEGER  
result.Download.ClientMeasurements.TCPInfo RECORD  
result.Download.ClientMeasurements.TCPInfo.State INTEGER  
result.Download.ClientMeasurements.TCPInfo.CAState INTEGER  
result.Download.ClientMeasurements.TCPInfo.Retransmits INTEGER  
result.Download.ClientMeasurements.TCPInfo.Probes INTEGER  
result.Download.ClientMeasurements.TCPInfo.Backoff INTEGER  
result.Download.ClientMeasurements.TCPInfo.Options INTEGER  
result.Download.ClientMeasurements.TCPInfo.WScale INTEGER  
result.Download.ClientMeasurements.TCPInfo.AppLimited INTEGER  
result.Download.ClientMeasurements.TCPInfo.RTO INTEGER  
result.Download.ClientMeasurements.TCPInfo.ATO INTEGER  
result.Download.ClientMeasurements.TCPInfo.SndMSS INTEGER  
result.Download.ClientMeasurements.TCPInfo.RcvMSS INTEGER  
result.Download.ClientMeasurements.TCPInfo.Unacked INTEGER  
result.Download.ClientMeasurements.TCPInfo.Sacked INTEGER  
result.Download.ClientMeasurements.TCPInfo.Lost INTEGER  
result.Download.ClientMeasurements.TCPInfo.Retrans INTEGER  
result.Download.ClientMeasurements.TCPInfo.Fackets INTEGER  
result.Download.ClientMeasurements.TCPInfo.LastDataSent INTEGER  
result.Download.ClientMeasurements.TCPInfo.LastAckSent INTEGER  
result.Download.ClientMeasurements.TCPInfo.LastDataRecv INTEGER  
result.Download.ClientMeasurements.TCPInfo.LastAckRecv INTEGER  
result.Download.ClientMeasurements.TCPInfo.PMTU INTEGER  
result.Download.ClientMeasurements.TCPInfo.RcvSsThresh INTEGER  
result.Download.ClientMeasurements.TCPInfo.RTT INTEGER  
result.Download.ClientMeasurements.TCPInfo.RTTVar INTEGER  
result.Download.ClientMeasurements.TCPInfo.SndSsThresh INTEGER  
result.Download.ClientMeasurements.TCPInfo.SndCwnd INTEGER  
result.Download.ClientMeasurements.TCPInfo.AdvMSS INTEGER  
result.Download.ClientMeasurements.TCPInfo.Reordering INTEGER  
result.Download.ClientMeasurements.TCPInfo.RcvRTT INTEGER  
result.Download.ClientMeasurements.TCPInfo.RcvSpace INTEGER  
result.Download.ClientMeasurements.TCPInfo.TotalRetrans INTEGER  
result.Download.ClientMeasurements.TCPInfo.PacingRate INTEGER  
result.Download.ClientMeasurements.TCPInfo.MaxPacingRate INTEGER  
result.Download.ClientMeasurements.TCPInfo.BytesAcked INTEGER  
result.Download.ClientMeasurements.TCPInfo.BytesReceived INTEGER  
result.Download.ClientMeasurements.TCPInfo.SegsOut INTEGER  
result.Download.ClientMeasurements.TCPInfo.SegsIn INTEGER  
result.Download.ClientMeasurements.TCPInfo.NotsentBytes INTEGER  
result.Download.ClientMeasurements.TCPInfo.MinRTT INTEGER  
result.Download.ClientMeasurements.TCPInfo.DataSegsIn INTEGER  
result.Download.ClientMeasurements.TCPInfo.DataSegsOut INTEGER  
result.Download.ClientMeasurements.TCPInfo.DeliveryRate INTEGER  
result.Download.ClientMeasurements.TCPInfo.BusyTime INTEGER  
result.Download.ClientMeasurements.TCPInfo.RWndLimited INTEGER  
result.Download.ClientMeasurements.TCPInfo.SndBufLimited INTEGER  
result.Download.ClientMeasurements.TCPInfo.Delivered INTEGER  
result.Download.ClientMeasurements.TCPInfo.DeliveredCE INTEGER  
result.Download.ClientMeasurements.TCPInfo.BytesSent INTEGER  
result.Download.ClientMeasurements.TCPInfo.BytesRetrans INTEGER  
result.Download.ClientMeasurements.TCPInfo.DSackDups INTEGER  
result.Download.ClientMeasurements.TCPInfo.ReordSeen INTEGER  
result.Download.ClientMeasurements.TCPInfo.ElapsedTime INTEGER  
result.Download.ClientMetadata RECORD  
result.Download.ClientMetadata.Name STRING  
result.Download.ClientMetadata.Value STRING  
Back to Top