NDT measures “bulk transport capacity”: the maximum date rate that TCP can reliably deliver data using the unreliable IP protocol over an end-to-end Internet path. TCP’s goal is to send data at exactly the optimal rate for the network, containing just the right mix of new data and retransmissions such that the receiver gets exactly one copy of everything. Since its creation, the TCP protocol has consistently made improvements to the way it accomplishes this task, consequently, NDT has also incrementally changed to reflect these improvements. The most recent improvements, including support for TCP BBR, are available in ndt7. On July 24th, we announced the start of migration of NDT clients to the latest protocol version. As of today, approximately 50% of clients are using ndt7. As the ndt7 measurements become the majority of the NDT dataset, the M-Lab team is considering what we do and do not know about whether and how changes to the NDT protocol have affected M-Lab’s longitudinal NDT dataset over time.
Following the general availability of the ndt7 protocol, we will be working with NDT client integrators to support their migration to ndt7. As they do, the NDT dataset will shift from predominantly ndt5 to predominantly ndt7. As part of assessing our readiness for this larger effort, a pilot was started on July 8.
The new ndt7 protocol for the Network Diagnostic Tool (NDT) is now generally available on the M-Lab platform. Since 2009, NDT has been the premier TCP performance measurement service test hosted by M-Lab. During its history on the platform, NDT has produced the largest test volume to date, spanning the longest history. Since late 2018, M-Lab has worked with researcher Simone Basso to develop the ndt7 protocol and archival data format.
When the M-Lab platform was initially launched in 2009, the software and operating system running on our servers used the best available boot management, virtualization, and kernel-level measurement instrumentation available. In the years since M-Lab’s initial launch, the state of system administration has improved dramatically. In 2017, the M-Lab team began work to upgrade the platform to adopt modern and flexible system administration components. This post provides a roadmap of that work.
What is TCP?
Since the beginning of the its existence, the Internet has expanded in scope, traffic, content, and a myriad of other ways. The protocols that make up the Internet’s backbone have mostly remained the same since they were developed in the 1980s. The Transmission Control Protocol (TCP) was one of the first networking protocols defined during the Internet’s development, and specifies how data should be transmitted and received. TCP implementations, initially developed in the 1980’s, attempted to discover the right rate at which to send data by constantly trying to send more until reaching the point that not all of the data arrived at its destination, and then backing off on the amount being sent.