web100 - NDT (Network Diagnostic Tool)

web100 is the NDT protocol currently used by M-Lab. As a part of M-Lab’s platform upgrade to be completed by the end of 2019, ndt-server will replace the current web100-based NDT server, provide the ndt5 protocol for backward compatibility with current clients, and will add the ndt7 protocol for future client use.

NDT data using the web100 protocol is collected using the Web100 Linux kernel patch to provide access to a rich set of TCP information for each NDT test.

web100 NDT data can be found in the NDT web100 dataset.

Equivalent BigQuery and Web100 Field Types

The file tcp-kis.txt defines each Web100 variable with a specific SNMP type. The table below shows how to map each SNMP type to a BigQuery type.

BigQuery Type Corresponding SNMP Type
integer Integer32, Integer, INTEGER, Gauge32, ZeroBasedCounter32, Unsigned32, Unsigned16, Counter32, ZeroBasedCounter64
string Ip_Address
bool TruthValue

Blacklist Flags Field

The field blacklist_flags was used to mark test affected by the “switch discard issue” identified in 2015-2016. M-Lab NDT data from 2010-01-01 to 2015-10-02 was marked using this field. M-Lab may use the field for other use cases in the future for tests potentially impacted by site configuration issues, or otherwise communicate potentially relevant information about the state of the platform at the time of the test.

Currently, the following values are present in NDT data from 2010-01-01 to 2015-10-02 in these fields:

Field Name Value Description
blacklist_flags or anomalies.blacklist_flags 0 or NULL unaffected tests
  1 tests affected by switch discards
  2 tests not shown to be unaffected by switch discards

web100 NDT BigQuery “Faithful” Schema

Field name Type Description
partition_date DATE  
test_id STRING  
task_filename STRING  
parse_time TIMESTAMP  
parser_version STRING  
log_time TIMESTAMP  
blacklist_flags INTEGER  
anomalies RECORD  
anomalies. no_meta BOOLEAN  
anomalies. snaplog_error BOOLEAN  
anomalies. num_snaps INTEGER  
anomalies. blacklist_flags INTEGER  
connection_spec RECORD  
connection_spec. client_af INTEGER  
connection_spec. client_application STRING  
connection_spec. client_browser STRING  
connection_spec. client_hostname STRING  
connection_spec. client_ip STRING  
connection_spec. client_kernel_version STRING  
connection_spec. client_os STRING  
connection_spec. client_version STRING  
connection_spec. data_direction INTEGER  
connection_spec. server_af INTEGER  
connection_spec. server_hostname STRING  
connection_spec. server_ip STRING  
connection_spec. server_kernel_version STRING  
connection_spec. tls BOOLEAN  
connection_spec. websockets BOOLEAN  
connection_spec. client_geolocation RECORD  
connection_spec.client_geolocation. area_code INTEGER  
connection_spec.client_geolocation. city STRING  
connection_spec.client_geolocation. continent_code STRING  
connection_spec.client_geolocation. country_code STRING  
connection_spec.client_geolocation. country_code3 STRING  
connection_spec.client_geolocation. country_name STRING  
connection_spec.client_geolocation. latitude FLOAT  
connection_spec.client_geolocation. longitude FLOAT  
connection_spec.client_geolocation. metro_code INTEGER  
connection_spec.client_geolocation. postal_code STRING  
connection_spec.client_geolocation. region STRING  
connection_spec.client_geolocation. radius INTEGER  
connection_spec. server_geolocation RECORD  
connection_spec.server_geolocation. area_code INTEGER  
connection_spec.server_geolocation. city STRING  
connection_spec.server_geolocation. continent_code STRING  
connection_spec.server_geolocation. country_code STRING  
connection_spec.server_geolocation. country_code3 STRING  
connection_spec.server_geolocation. country_name STRING  
connection_spec.server_geolocation. latitude FLOAT  
connection_spec.server_geolocation. longitude FLOAT  
connection_spec.server_geolocation. metro_code INTEGER  
connection_spec.server_geolocation. postal_code STRING  
connection_spec.server_geolocation. region STRING  
connection_spec.server_geolocation. radius INTEGER  
connection_spec. client RECORD  
connection_spec.client. network RECORD  
connection_spec.client.network. asn STRING  
connection_spec. server RECORD  
connection_spec.server. iata_code STRING  
connection_spec.server. network RECORD  
connection_spec.server.network. asn STRING  
web100_log_entry RECORD  
web100_log_entry. log_time INTEGER  
web100_log_entry. version STRING  
web100_log_entry. connection_spec RECORD  
web100_log_entry.connection_spec. local_af INTEGER  
web100_log_entry.connection_spec. local_ip STRING  
web100_log_entry.connection_spec. local_port INTEGER  
web100_log_entry.connection_spec. remote_ip STRING  
web100_log_entry.connection_spec. remote_port INTEGER  
web100_log_entry. snap RECORD  
web100_log_entry.snap. AbruptTimeouts INTEGER  
web100_log_entry.snap. ActiveOpen INTEGER  
web100_log_entry.snap. CERcvd INTEGER  
web100_log_entry.snap. CongAvoid INTEGER  
web100_log_entry.snap. CongOverCount INTEGER  
web100_log_entry.snap. CongSignals INTEGER  
web100_log_entry.snap. CountRTT INTEGER  
web100_log_entry.snap. CurAppRQueue INTEGER  
web100_log_entry.snap. CurAppWQueue INTEGER  
web100_log_entry.snap. CurCwnd INTEGER  
web100_log_entry.snap. CurMSS INTEGER  
web100_log_entry.snap. CurRTO INTEGER  
web100_log_entry.snap. CurReasmQueue INTEGER  
web100_log_entry.snap. CurRetxQueue INTEGER  
web100_log_entry.snap. CurRwinRcvd INTEGER  
web100_log_entry.snap. CurRwinSent INTEGER  
web100_log_entry.snap. CurSsthresh INTEGER  
web100_log_entry.snap. CurTimeoutCount INTEGER  
web100_log_entry.snap. DSACKDups INTEGER  
web100_log_entry.snap. DataSegsIn INTEGER  
web100_log_entry.snap. DataSegsOut INTEGER  
web100_log_entry.snap. DupAcksIn INTEGER  
web100_log_entry.snap. DupAcksOut INTEGER  
web100_log_entry.snap. Duration INTEGER  
web100_log_entry.snap. ECN INTEGER  
web100_log_entry.snap. FastRetran INTEGER  
web100_log_entry.snap. HCDataOctetsIn INTEGER  
web100_log_entry.snap. HCDataOctetsOut INTEGER  
web100_log_entry.snap. HCThruOctetsAcked INTEGER  
web100_log_entry.snap. HCThruOctetsReceived INTEGER  
web100_log_entry.snap. LimCwnd INTEGER  
web100_log_entry.snap. LimRwin INTEGER  
web100_log_entry.snap. LocalAddress STRING  
web100_log_entry.snap. LocalAddressType INTEGER  
web100_log_entry.snap. LocalPort INTEGER  
web100_log_entry.snap. MSSRcvd INTEGER  
web100_log_entry.snap. MaxAppRQueue INTEGER  
web100_log_entry.snap. MaxAppWQueue INTEGER  
web100_log_entry.snap. MaxMSS INTEGER  
web100_log_entry.snap. MaxRTO INTEGER  
web100_log_entry.snap. MaxRTT INTEGER  
web100_log_entry.snap. MaxReasmQueue INTEGER  
web100_log_entry.snap. MaxRetxQueue INTEGER  
web100_log_entry.snap. MaxRwinRcvd INTEGER  
web100_log_entry.snap. MaxRwinSent INTEGER  
web100_log_entry.snap. MaxSsCwnd INTEGER  
web100_log_entry.snap. MaxSsthresh INTEGER  
web100_log_entry.snap. MinMSS INTEGER  
web100_log_entry.snap. MinRTO INTEGER  
web100_log_entry.snap. MinRTT INTEGER  
web100_log_entry.snap. MinRwinRcvd INTEGER  
web100_log_entry.snap. MinRwinSent INTEGER  
web100_log_entry.snap. MinSsthresh INTEGER  
web100_log_entry.snap. Nagle INTEGER  
web100_log_entry.snap. NonRecovDA INTEGER  
web100_log_entry.snap. OctetsRetrans INTEGER  
web100_log_entry.snap. OtherReductions INTEGER  
web100_log_entry.snap. PostCongCountRTT INTEGER  
web100_log_entry.snap. PostCongSumRTT INTEGER  
web100_log_entry.snap. PreCongSumCwnd INTEGER  
web100_log_entry.snap. PreCongSumRTT INTEGER  
web100_log_entry.snap. QuenchRcvd INTEGER  
web100_log_entry.snap. RTTVar INTEGER  
web100_log_entry.snap. RcvNxt INTEGER  
web100_log_entry.snap. RcvRTT INTEGER  
web100_log_entry.snap. RcvWindScale INTEGER  
web100_log_entry.snap. RecInitial INTEGER  
web100_log_entry.snap. RemAddress STRING  
web100_log_entry.snap. RemPort INTEGER  
web100_log_entry.snap. RetranThresh INTEGER  
web100_log_entry.snap. SACK INTEGER  
web100_log_entry.snap. SACKBlocksRcvd INTEGER  
web100_log_entry.snap. SACKsRcvd INTEGER  
web100_log_entry.snap. SampleRTT INTEGER  
web100_log_entry.snap. SegsIn INTEGER  
web100_log_entry.snap. SegsOut INTEGER  
web100_log_entry.snap. SegsRetrans INTEGER  
web100_log_entry.snap. SendStall INTEGER  
web100_log_entry.snap. SlowStart INTEGER  
web100_log_entry.snap. SmoothedRTT INTEGER  
web100_log_entry.snap. SndInitial INTEGER  
web100_log_entry.snap. SndLimBytesCwnd INTEGER  
web100_log_entry.snap. SndLimBytesRwin INTEGER  
web100_log_entry.snap. SndLimBytesSender INTEGER  
web100_log_entry.snap. SndLimTimeCwnd INTEGER  
web100_log_entry.snap. SndLimTimeRwin INTEGER  
web100_log_entry.snap. SndLimTimeSnd INTEGER  
web100_log_entry.snap. SndLimTransCwnd INTEGER  
web100_log_entry.snap. SndLimTransRwin INTEGER  
web100_log_entry.snap. SndLimTransSnd INTEGER  
web100_log_entry.snap. SndMax INTEGER  
web100_log_entry.snap. SndNxt INTEGER  
web100_log_entry.snap. SndUna INTEGER  
web100_log_entry.snap. SndWindScale INTEGER  
web100_log_entry.snap. SpuriousFrDetected INTEGER  
web100_log_entry.snap. StartTimeStamp INTEGER  
web100_log_entry.snap. StartTimeUsec INTEGER  
web100_log_entry.snap. State INTEGER  
web100_log_entry.snap. SubsequentTimeouts INTEGER  
web100_log_entry.snap. SumRTT INTEGER  
web100_log_entry.snap. TimeStamps INTEGER  
web100_log_entry.snap. Timeouts INTEGER  
web100_log_entry.snap. WinScaleRcvd INTEGER  
web100_log_entry.snap. WinScaleSent INTEGER  
web100_log_entry.snap. X_OtherReductionsCM INTEGER  
web100_log_entry.snap. X_OtherReductionsCV INTEGER  
web100_log_entry.snap. X_Rcvbuf INTEGER  
web100_log_entry.snap. X_Sndbuf INTEGER  
web100_log_entry.snap. X_dbg1 INTEGER  
web100_log_entry.snap. X_dbg2 INTEGER  
web100_log_entry.snap. X_dbg3 INTEGER  
web100_log_entry.snap. X_dbg4 INTEGER  
web100_log_entry.snap. X_rcv_ssthresh INTEGER  
web100_log_entry.snap. X_wnd_clamp INTEGER  
web100_log_entry. deltas RECORD REPEATED  
web100_log_entry.deltas. is_last BOOLEAN  
web100_log_entry.deltas. snapshot_num INTEGER  
web100_log_entry.deltas. delta_index INTEGER  
web100_log_entry.deltas. AbruptTimeouts INTEGER  
web100_log_entry.deltas. ActiveOpen INTEGER  
web100_log_entry.deltas. CERcvd INTEGER  
web100_log_entry.deltas. CongAvoid INTEGER  
web100_log_entry.deltas. CongOverCount INTEGER  
web100_log_entry.deltas. CongSignals INTEGER  
web100_log_entry.deltas. CountRTT INTEGER  
web100_log_entry.deltas. CurAppRQueue INTEGER  
web100_log_entry.deltas. CurAppWQueue INTEGER  
web100_log_entry.deltas. CurCwnd INTEGER  
web100_log_entry.deltas. CurMSS INTEGER  
web100_log_entry.deltas. CurRTO INTEGER  
web100_log_entry.deltas. CurReasmQueue INTEGER  
web100_log_entry.deltas. CurRetxQueue INTEGER  
web100_log_entry.deltas. CurRwinRcvd INTEGER  
web100_log_entry.deltas. CurRwinSent INTEGER  
web100_log_entry.deltas. CurSsthresh INTEGER  
web100_log_entry.deltas. CurTimeoutCount INTEGER  
web100_log_entry.deltas. DSACKDups INTEGER  
web100_log_entry.deltas. DataSegsIn INTEGER  
web100_log_entry.deltas. DataSegsOut INTEGER  
web100_log_entry.deltas. DupAcksIn INTEGER  
web100_log_entry.deltas. DupAcksOut INTEGER  
web100_log_entry.deltas. Duration INTEGER  
web100_log_entry.deltas. ECN INTEGER  
web100_log_entry.deltas. FastRetran INTEGER  
web100_log_entry.deltas. HCDataOctetsIn INTEGER  
web100_log_entry.deltas. HCDataOctetsOut INTEGER  
web100_log_entry.deltas. HCThruOctetsAcked INTEGER  
web100_log_entry.deltas. HCThruOctetsReceived INTEGER    
web100_log_entry.deltas. LimCwnd INTEGER  
web100_log_entry.deltas. LimRwin INTEGER  
web100_log_entry.deltas. MSSRcvd INTEGER  
web100_log_entry.deltas. MaxAppRQueue INTEGER  
web100_log_entry.deltas. MaxAppWQueue INTEGER  
web100_log_entry.deltas. MaxMSS INTEGER  
web100_log_entry.deltas. MaxRTO INTEGER  
web100_log_entry.deltas. MaxRTT INTEGER  
web100_log_entry.deltas. MaxReasmQueue INTEGER  
web100_log_entry.deltas. MaxRetxQueue INTEGER  
web100_log_entry.deltas. MaxRwinRcvd INTEGER  
web100_log_entry.deltas. MaxRwinSent INTEGER  
web100_log_entry.deltas. MaxSsCwnd INTEGER  
web100_log_entry.deltas. MaxSsthresh INTEGER  
web100_log_entry.deltas. MinMSS INTEGER  
web100_log_entry.deltas. MinRTO INTEGER  
web100_log_entry.deltas. MinRTT INTEGER  
web100_log_entry.deltas. MinRwinRcvd INTEGER  
web100_log_entry.deltas. MinRwinSent INTEGER  
web100_log_entry.deltas. MinSsthresh INTEGER  
web100_log_entry.deltas. Nagle INTEGER  
web100_log_entry.deltas. NonRecovDA INTEGER  
web100_log_entry.deltas. OctetsRetrans INTEGER  
web100_log_entry.deltas. OtherReductions INTEGER  
web100_log_entry.deltas. PostCongCountRTT INTEGER  
web100_log_entry.deltas. PostCongSumRTT INTEGER  
web100_log_entry.deltas. PreCongSumCwnd INTEGER  
web100_log_entry.deltas. PreCongSumRTT INTEGER  
web100_log_entry.deltas. QuenchRcvd INTEGER  
web100_log_entry.deltas. RTTVar INTEGER  
web100_log_entry.deltas. RcvNxt INTEGER  
web100_log_entry.deltas. RcvRTT INTEGER  
web100_log_entry.deltas. RcvWindScale INTEGER  
web100_log_entry.deltas. RecInitial INTEGER  
web100_log_entry.deltas. RetranThresh INTEGER  
web100_log_entry.deltas. SACKBlocksRcvd INTEGER  
web100_log_entry.deltas. SACKsRcvd INTEGER  
web100_log_entry.deltas. SampleRTT INTEGER  
web100_log_entry.deltas. SegsIn INTEGER  
web100_log_entry.deltas. SegsOut INTEGER  
web100_log_entry.deltas. SegsRetrans INTEGER  
web100_log_entry.deltas. SendStall INTEGER  
web100_log_entry.deltas. SlowStart INTEGER  
web100_log_entry.deltas. SmoothedRTT INTEGER  
web100_log_entry.deltas. SndInitial INTEGER  
web100_log_entry.deltas. SndLimBytesCwnd INTEGER  
web100_log_entry.deltas. SndLimBytesRwin INTEGER  
web100_log_entry.deltas. SndLimBytesSender INTEGER  
web100_log_entry.deltas. SndLimTimeCwnd INTEGER  
web100_log_entry.deltas. SndLimTimeRwin INTEGER  
web100_log_entry.deltas. SndLimTimeSnd INTEGER  
web100_log_entry.deltas. SndLimTransCwnd INTEGER  
web100_log_entry.deltas. SndLimTransRwin INTEGER  
web100_log_entry.deltas. SndLimTransSnd INTEGER  
web100_log_entry.deltas. SndMax INTEGER  
web100_log_entry.deltas. SndNxt INTEGER  
web100_log_entry.deltas. SndUna INTEGER  
web100_log_entry.deltas. SndWindScale INTEGER  
web100_log_entry.deltas. SpuriousFrDetected INTEGER  
web100_log_entry.deltas. StartTimeStamp INTEGER  
web100_log_entry.deltas. StartTimeUsec INTEGER  
web100_log_entry.deltas. State INTEGER  
web100_log_entry.deltas. SubsequentTimeouts INTEGER  
web100_log_entry.deltas. SumRTT INTEGER  
web100_log_entry.deltas. TimeStamps INTEGER  
web100_log_entry.deltas. Timeouts INTEGER  
web100_log_entry.deltas. WinScaleRcvd INTEGER  
web100_log_entry.deltas. WinScaleSent INTEGER  
web100_log_entry.deltas. X_OtherReductionsCM INTEGER  
web100_log_entry.deltas. X_OtherReductionsCV INTEGER  
web100_log_entry.deltas. X_Rcvbuf INTEGER  
web100_log_entry.deltas. X_Sndbuf INTEGER  
web100_log_entry.deltas. X_dbg1 INTEGER  
web100_log_entry.deltas. X_dbg2 INTEGER  
web100_log_entry.deltas. X_dbg3 INTEGER  
web100_log_entry.deltas. X_dbg4 INTEGER  
web100_log_entry.deltas. X_rcv_ssthresh INTEGER  
web100_log_entry.deltas. X_wnd_clamp INTEGER  
Back to Top