2.8.6 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 60th IETF Meeting in San Diego, CA USA. It may now be out-of-date.

Last Modified: 2004-07-23

Chair(s):
Henk Uijterwaal <henk@ripe.net>
Matthew Zekauskas <matt@internet2.edu>
Transport Area Director(s):
Allison Mankin <mankin@psg.com>
Jon Peterson <jon.peterson@neustar.biz>
Transport Area Advisor:
Allison Mankin <mankin@psg.com>
Technical Advisor(s):
Andy Bierman <abierman@cisco.com>
Mailing Lists:
General Discussion: ippm@ietf.org
To Subscribe: https://www1.ietf.org/mailman/listinfo/ippm
In Body: subscribe
Archive: http://www.ietf.org/mail-archive/web/ippm/index.html
Description of Working Group:
Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgment (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and procedures for accurately measuring and documenting these metrics. The metrics are:

- connectivity

- one-way delay and loss

- round-trip delay and loss

- delay variation

- loss patterns

- packet reordering

- bulk transport capacity

- link bandwidth capacity

This is the cumulative set, including the metricsalready completed and published.

The working group will closely review and then be guided by an IESG document on how metrics advance along the standards track within the IETF. This document will also be relevant to the work of the benchmarking working group (BMWG). The first draft of this document was discussed at IETF 51. Additionally, the WG will produce Proposed Standard AS documents, comparable to applicability statements in RFC 2026, that will focus on procedures for measuring the individual metrics and how these metrics characterize features that are important to different service classes, such as bulk transport, periodic streams, or multimedia streams. It is specifically out of scope for this working group to actually characterize traffic, for example to characterize a voice-over-IP stream. Each AS document will discuss the performance characteristics that are pertinent to a specified service class; clearly identify the set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of testing results. Specific topics of these AS documents must be approved by the Area Directors as charter additions.

The WG will produce a protocol to enable communication among test equipment that implements the one-way metrics. The intent is to create a protocol that provides a base level of functionality that will allow different manufacturer's equipment that implements the metrics according to a standard to interoperate. A protocol requirements document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM metrics, such as one-way delay and loss, to facilitate the communication of metrics to existing network management systems. Thus, the group will create a MIB that contains predominantly read only variables. If, after the protocol requirements document is finished, the group decides that it is appropriate to add variables that control the underlying measurements that the metrics report, such a control structure may be added as a separate document, subject to review by the IESG.

The intent of the WG is to cooperate with other appropriate standards bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote consistent approaches and metrics. Within the IETF process, IPPM metrics definitions will be subject to as rigorous a scrutiny for usefulness, clarity, and accuracy as other protocol standards. The IPPM WG will interact with other areas of IETF activity whose scope intersect with the requirement of these specific metrics. These include working groups such as BMWG, RMONMIB, and TEWG.

Goals and Milestones:
Done  Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done  Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done  Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done  Submit a document on connectivity to IESG for publication as an Informational RFC.
Done  Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
Done  Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
Done  Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
Done  Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
Done  First draft for AS on one-way delay and loss.
Done  Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
Done  Create initial draft on a MIB for reporting IPPM metrics.
Done  Create initial draft on a packet reordering metric.
Aug 04  Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
Aug 04  Create initial draft on the definitions of link bandwidth capacity.
Aug 04  Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
Oct 04  Submit draft on a packet reordering metric to the IESG for Proposed Standard.
Dec 04  Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard.
Dec 04  Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
Dec 04  Collect implementation reports for RFCs 2678-2681
Feb 05  Discuss rechartering or ending working group.
Internet-Drafts:
  • - draft-ietf-ippm-owdp-09.txt
  • - draft-ietf-ippm-metrics-registry-07.txt
  • - draft-ietf-ippm-reporting-mib-06.txt
  • - draft-ietf-ippm-reordering-06.txt
  • Request For Comments:
    RFCStatusTitle
    RFC2330 I Framework for IP Performance Metrics
    RFC2678 E IPPM Metrics for Measuring Connectivity
    RFC2679 PS A One-way Delay Metric for IPPM
    RFC2680 PS A One-way Packet Loss Metric for IPPM
    RFC2681 PS A Round-trip Delay Metric for IPPM
    RFC3148 I A Framework for Defining Empirical Bulk Transfer Capacity Metrics
    RFC3357 I One-way Loss Pattern Sample Metrics
    RFC3393 PS IP Packet Delay Variation Metric for IPPM
    RFC3432 PS Network performance measurement for periodic streams
    RFC3763 I A One-way Active Measurement Protocol Requirements

    Current Meeting Report

    IP Performance Metrics WG (ippm)
    Monday, August 2, 2004 at 1300-1500
    ===================================
    The meeting was moderated by the working group chairs, Henk Uijterwaal and Matt Zekauskas. Al Morton and Matt Zekauskas took notes, which were edited into these minutes by the chairs. Simon Leinen scribed to Jabber.


    AGENDA:


    1. Administrivia, Milestone Status
    2. One-Way Active Measurements Protocol
    3. A Hardware Timestamper for One-way Delay Measurements
    4. Reordering Density (RD) and Reorder Buffer-occupancy Density (RBD): Metrics for packet reordering
    5. Packet Reordering Metric for IPPM
    6. Implementation reports for RFC2678-2681: RIPE NCC Test Traffic Measurements (TTM)
    7. Implementation report: Surveyor
    8. IPPM Reporting MIB and Metrics Registry discussion




    1. Administrivia, Milestone Status

    We are looking for a volunteer to act as editor on link bandwidth draft, there is some existing material to base it on. [Note: after the meeting, Phil Chimento volunteered to edit.] Reviewing the rest of the milestones, OWAMP appears to be on track, reordering schedule will depend on today's discussion.




    2. One-Way Active Measurements Protocol
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-09.txt
    - Stanislav Shalunov

    There were two wire protocol changes and a bunch of clarifications since the last version (see slides). Changes were made in response to the approved requirements draft, two wire protocol changes (to expose TTL, and a minor change in reporting lost packets, both in response to implementation experience). Stanislav reported that there were pending changes to remove a feature that caused some ambiguity (essentially "start in the past"), and a reordering of values in one message to make it more likely that the timestamp falls on a nice word boundary. Stanislav mentioned that the protocol really needed a thorough security review.

    TTL is intended to count hops, was not specified before, now receiver must report actual received value (sender uses 255). Matt asked why 255? Why not negotiated? Stanislav said that it made sense to have it large and equal. Timmons Player asked what if an implementation cannot set the value? Stanislav said that then you get back what you get back ("junk"); it should be possible to report if you know what it is set to (so you can understand in context).

    Al Morton asked a question on send scheduling for test packets. He was wondering if there was a way to have a random start time, then a periodic schedule after that. It seems that the current scheme would try to do random, periodic, and then back to random again. Stas believed it could be done with the current scheme; Stas and Al will work on this afterwards.




    3. A Hardware Timestamper for One-way Delay Measurements
    - Zhang Shu

    Zhang Shu reported on a hardware implementation of OWAMP, in particular, timestamping the packets in hardware, to get rid of inaccuracies due to software overhead and timing within the measurement node. The limitations are that it supports unauthenticated mode only, one measurement stream only, and on INPUT the UDP checksum is zeroed (when the input timestamp is added). They showed the results of measurements in Japan using IPv4 and IPv6. They claim high precision (see slides).

    Stanislav thought this was splendid. He noted that they apparently did not make use of OWAMP-CTL anywhere. That is true. So, they should make it clear that they use the test versus the full OWAMP protocol.

    Matt asked about support of more than one port number or stream? not yet. Matt also asked if before the UDP checksum was cleared, was it checked? Right now, it is cleared. Zhang felt that in the current internet there isn't much chance for this to happen.

    Emile Stephan asked how the test endpoints were set up without OWAMP-control. Zhang responded that setup was performed manually.




    4. Reordering Density (RD) and Reorder Buffer-occupancy Density (RBD):
    Metrics for packet reordering
    http://www.ietf.org/internet-drafts/draft-jayasumana-reorder-density-03.txt
    - Anura Jayasumana

    Anura Jayasumana presented an update on his team's work on "reorder density" (see slides). He presented a new version of reorder density that he says is orthogonal to loss and duplication, but it still doesn't use the same base definitions as in the current IPPM reordering draft. He says that this provides a "reorder response" similar to "impulse response" in engineering control theory, which has a formal proof in a recent paper; they are looking to use this as a base for a version of TCP that sets "dupthresh" automatically. He also said the results on individual segments are composable. He may be interested in submitting this as an additional work item, or move it forward personally if the group is not interested.

    Mark Allman asked what do the numbers output by these metrics say? He didn't think the interpretation was obvious; how do you get insight into how the network is working from the numbers? Anura responded that it gives insight into the nature of the reordering; they give a distribution.

    Al Morton thought the real value was the ability to measure segments and combine. This is a desirable quality in current IPPM reordering draft. However, intuitively he doesn't believe that these distributions are independent of packet spacing. Al would like to harmonize basic definition of reordering (so same as what current draft is, and what others use).

    Anura said that to understand the independence of spacing, look at system theory. You are not predicting the effect on any given packet, but statistical nature. With respect to the current ippm draft, The current definitions only work with "early" or "late" packets, don't combine the two. You need this combination in order to get the insight.




    5. Packet Reordering Metric for IPPM
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-reordering-06.txt
    - Al Morton

    Al Morton presented the current state of the IPPM reordering draft. The fragmentation additions were complex enough to be confusing in the mainline text, so the discussion has been moved to the appendix. There were many clarification changes based on comments on the list, off the list, and at meetings (see slides). Two new and independent groups have implemented the metric (and there was some discussion on the list).

    Al asked the meeting if there were further comments, and there were none.




    6. Implementation reports for RFC2678-2681:
    RIPE NCC Test Traffic Measurements (TTM)

    Henk presented an template for producing an implementation report, including what was asked for at the last IETF meeting and on the mailing list. The purpose of these presentations are to provide complete examples, and encourage other implementers to contribute.


    For 2679, all metrics were implemented except Type-P-One-way-Delay-Inverse-Percentile. For 2680, all were implemented. 2678 (connectivity) and 2681 (round trip delay) were not implemented by this system. Specific percentiles and averaging intervals used were given.




    7. Implementation report: Surveyor

    Matt Zekauskas made a similar report for Surveyor. For 2679, all were implemented except the inverse percentile metric (although histograms of delays were computed). Matt gave more details on the "Type-P" used, as well as transmission rates. He also gave percentiles and averaging intervals used. As with the RIPE system Surveyor implemented all of the 2680 metrics, and non of 2678 or 2681.




    8. IPPM Reporting MIB and Metrics Registry discussion
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-reporting-mib-06.txt
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-metrics-registry-07.txt
    - Emile Stephan

    Emile noted that there was a move to stop work on the MIB as a working group item. He also noted that in other groups he observed that there wasn't much discussion on the respective groups. He then noted what made the IPPM MIB different from some others, and presented a list of questions the group. He also briefly updated the latest changes.


    Matt presented the chairs position that there appears to be no consensus to continue with the current document, and that the plan will be to once again query the mailing list looking for interest in either this MIB, or the simpler MIB; if none then we would drop this item from the charter (and Emile would be free to request publication as an Informational RFC).


    Andy Bierman noted that perhaps because the group was interested in measurement, not management (defining metrics, not MIBs) there was little interest in the MIB. Andy said that it is a very complicated MIB, and it needs better use cases. The packet generation is not typical. The idea of fast report is not typical. There is some overlap with TPM MIB, for some of the metrics. Also, once you get into views, filtering, aggregation, something like TPM already does it. He's of the opinion that we should not proceed with the MIB.


    Phil Chimento asked if it made sense to have a general MIB at all, or is it better to have reporting in the context of an application? Andy agreed to the need to provide results in the context of an application.


    There was some short discussion of the pros and cons of a simple ring-buffer based MIB, and whether the update rates would just overwhelm a simple implementation. Was there a use case for a simple MIB?


    Finally, Matt presented the remaining open issue with the MIB registry, whether to include branches for enterprises and "other standards bodies". Andy was of the opinion that a centralized place to look for the values is enough.

    Slides

    Agenda
    One-Way Active Measurement Protocol
    A Hardware Timestamper for One-Way Delay Measurements
    Reorder Density (RD) and Reorder Buffer-occupancy Density (RBD) : Metrics for packet reordering
    Packet Reordering Metric for IPPM
    Implementation reports
    Surveyor Implementation Report
    IPPM REPORTING MIB
    On the IPPM MIB
    IPPM Metrics Registry