NOTE: This charter is a snapshot of the 41st IETF Meeting in Los Angeles, California. It may now be out-of-date. Last Modified: 11-Mar-98
Chair(s):
Guy Almes <almes@advanced.org>
Vern Paxson <vern@ee.lbl.gov>
Transport Area Director(s):
Scott Bradner <sob@harvard.edu>
Allyn Romanow <allyn@mci.net>
Transport Area Advisor:
Scott Bradner <sob@harvard.edu>
Mailing Lists:
General Discussion:ippm@advanced.org
To Subscribe: ippm-request@advanced.org
Archive: ftp://ftp.advanced.org/pub/IPPM/archive
Description of Working Group:
The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgement (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.
Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.
The IPPM WG will define specific metrics, cultivate technology for the accurate measurement and documentation of these metrics, and promote the sharing of effective tools and procedures for measuring these metrics. It will also offer a forum for sharing information about the implementation and application of these metrics, but actual implementations and applications are understood to be beyond the scope of this working group.
Goals and Milestones:
Oct 97 |
|
Submit drafts of standard metrics for connectivity and treno-bulk-throughput. |
Nov 97 |
|
Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC. |
Feb 98 |
|
Submit documents on delay and loss to IESG for publication as Informational RFCs. |
Apr 98 |
|
Submit a document on connectivity to IESG for publication as an Informational RFC. |
Sep 98 |
|
Submit a document on bulk-throughput to IESG for publication as an Informational RFC. |
Internet-Drafts:
· Framework for IP Performance Metrics
· A One-way Delay Metric for IPPM
· A Packet Loss Metric for IPPM
· Connectivity
· Instantaneous Packet Delay Variation Metric for IPPM
No Request For Comments
Minutes of the IP Performance Metrics (ippm) Working Group
1. Overview and Agenda
The meeting was chaired by WG co-chairs Guy Almes and Vern Paxson, and was well attended. The minutes were drafted from notes taken by Paul Love.
Agenda
1. Overview and Agenda
2. State of Framework Document
3. Issues in One-way Delay/Loss Metrics
4. Audio and Loss Patterns
5. Internet Delay Measurement using Test Traffic
6. Instantaneous Packet Delay Variation
The agenda were agreeable to the participants.
2. State of the Framework Document - Vern Paxson, Lawrence Berkeley Labs
Vern reported that the Framework document is now approved by IESG for publication as an Informational RFC. It is now in the production process.
3. Issues in One-way Delay/Loss Metrics - Guy Almes, Advanced Network and Services
Guy reported on three issues that relate to the Internet Drafts on One-way Delay and Packet Loss.
3.a. Wire Time versus System Time Revisited
In the Framework document, it is encouraged that metric definitions and implementations be cast in terms of wire time. For example, in the case of one-way delay, you would want to measure the time at which the packet hits the network rather than the time at which the host starts trying to send the packet.
Guy pointed out that in a contention-based shared access network such as the original CSMA/CD Ethernet, this may not be exactly what you would want to do. In a CSMA/CD Ethernet, the time spent by the host in contending for access to the Ethernet media can actually be viewed as queuing delay within the network.
Guy proposes no change to the Framework at the present, though. It seems to be a strong trend that a number of important network technologies are essentially point-to-point switched technologies, and wire time is just the right notion for them.
3.b. Calibration of One-way Delay and Packet Loss Implementations
Guy proposed that simple error bars, qualified by a 95% confidence interval, be used to characterize a measurement of one-way delay. This proposal led to a useful discussion. Part of the discussion related to the notions of resolution and accuracy of clocks. Vern noted that it would be important to point out how limits on the resolution and accuracy of the clocks at the source and at the destination would combine to characterize the inaccuracy of the one-way delay measurement.
Vern also noted that this would often lead to asymmetric error bars. He also noted that, at least in principle, error bars could lead to the possibility of negative delays (as when a measurement of 30 usec one-way delay across a LAN with 50 usec error bars). One participant noted that the error bars themselves might vary with time, though they would have durable upper bounds. It was also noted that it's generally bad form to claim a measurement with finer resolution than your accuracy. Thus, stating a measurement of 23.4653 msec would be bad communication if you had 100 usec error bars.
Brian Carpenter suggested that we make a distinction between errors from systematic and statistical sources. This is common practice in experimental physics and we may want to tap into a good source of expertise to apply this idea. The authors will attempt to learn more about this idea.
Vern repeated that a careful description of how to combine an understanding of various forms of clock errors/uncertainties would be an important part of a calibration section.
Will Leland cautioned to keep the ambition level appropriate for our needs.
Guy concluded that the likely outcome would be asymmetric error bars with a description of how to combine clock errors/uncertainties.
Finally, Guy noted a couple of points on the calibration of Packet Loss:
· Any implementation of packet loss should clarify what threshold of delay is used above which packets are regarded as lost.
· With both delay and packet loss, there is the issue of calibrating the timestamp at which the measurement occurs.
3.c. The Pesky Path Parameter
Guy noted that, over the lifetime of the various one-way delay Internet Drafts, three ideas have been discussed for including the path taken by a test packet from source to destination in the metric definition:
· make the complete path be a parameter of the metric. This is in the current Internet Draft. It's nice in principle, but it's almost impossible to have complete knowledge of both the path and the one-way delay.
· make the first-hop only be a parameter of the metric. This was in the first draft of the Internet Draft, but seems to be more of a complication that it's worth.
· remove the parameter entirely, so that source and destination IP addresses are the only parameters that relate to the path at all.
Guy recommended removing the path parameter entirely, and discussion proceeded of these options. Vern suggested that we remove the path parameter, but find a way to give guidance to implementors that, when they have useful information about the path, they include it. Will Leland noted that such optional path information might be part of a more general notion of context information for a given measurement. Brian Carpenter cautioned on allowing the context information to grow too general and thus becoming too vague to be useful.
4. Audio and Loss Patterns - Koodli Rajeev of Nokia Research Labs
Rajeev gave a presentation with sound demos showing different types of loss, motivating the need to measure them. By going beyond simple notions of percentage of packet loss over large time intervals and establishing notions of time-series metrics of packet loss, they are able to talk about the impact of patterns of loss on streaming applications. Rajeev demonstrated these notions by playing several copies of a Louis Armstrong song, each with a different kind of packet loss phenomenon. Discussion followed, highlighting the question of whether the effect of different forms of loss on audio perception has already been studied in some other forum, such as telephony.
5. Internet Delay Measurement Using Test Traffic - Olaf M. Kolkman of RIPE NCC
Olaf described an implementation of One-way Delay done at the RIPE NCC. This implementation is characterized by use of a PPS GPS signal to govern the system clock, and by effective use of the Berkeley Packet Filter. Calibration work has been done with good results. URL: http://www.ripe.net/test-traffic and http://www.ripe.net/test-traffic/Talks/9803_ietf/index.html
Guy noted that this is the third active independent implementation project. This bodes well for the metrics having solid independent implementations.
6. Instantaneous Packet Delay Variation - Carlo Demichelis from CSELT - Italy
Carlo discussed how to measure variation in one-way delay without the very tight GPS-enabled synchronization needed for measuring one-way delay. Vern asked several questions about the impact on accuracy if successive packets have variable delay between them. Carlos stressed that this work is not finished and includes research issues.
Issues in the One-way Delay and Packet Loss Metrics - Background and Open Issues
Some Experiments on Impact of Loss Pattern