Current Meeting Report
Slides


2.8.4 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 54th IETF Meeting in Yokohama, Japan. It may now be out-of-date.

Last Modifield: 05/06/2002

Chair(s):
Merike Kaeo <kaeo@merike.com>
Matthew Zekauskas <matt@advanced.org>
Transport Area Director(s):
Scott Bradner <sob@harvard.edu>
A. Mankin <mankin@isi.edu>
Transport Area Advisor:
A. Mankin <mankin@isi.edu>
Technical Advisor(s):
Andy Bierman <abierman@cisco.com>
Mailing Lists:
General Discussion: ippm@advanced.org
To Subscribe: ippm-request@advanced.org
Archive: http://www.advanced.org/IPPM/archive/
Description of Working Group:
Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgment (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and procedures for accurately measuring and documenting these metrics. The metrics are:

- connectivity

- one-way delay and loss

- round-trip delay and loss

- delay variation

- loss patterns

- packet reordering

- bulk transport capacity

- link bandwidth capacity

This is the cumulative set, including the metricsalready completed and published.

The working group will closely review and then be guided by an IESG document on how metrics advance along the standards track within the IETF. This document will also be relevant to the work of the benchmarking working group (BMWG). The first draft of this document was discussed at IETF 51. Additionally, the WG will produce Proposed Standard AS documents, comparable to applicability statements in RFC 2026, that will focus on procedures for measuring the individual metrics and how these metrics characterize features that are important to different service classes, such as bulk transport, periodic streams, or multimedia streams. It is specifically out of scope for this working group to actually characterize traffic, for example to characterize a voice-over-IP stream. Each AS document will discuss the performance characteristics that are pertinent to a specified service class; clearly identify the set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of testing results. Specific topics of these AS documents must be approved by the Area Directors as charter additions.

The WG will produce a protocol to enable communication among test equipment that implements the one-way metrics. The intent is to create a protocol that provides a base level of functionality that will allow different manufacturer's equipment that implements the metrics according to a standard to interoperate. A protocol requirements document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM metrics, such as one-way delay and loss, to facilitate the communication of metrics to existing network management systems. Thus, the group will create a MIB that contains predominantly read only variables. If, after the protocol requirements document is finished, the group decides that it is appropriate to add variables that control the underlying measurements that the metrics report, such a control structure may be added as a separate document, subject to review by the IESG.

The intent of the WG is to cooperate with other appropriate standards bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote consistent approaches and metrics. Within the IETF process, IPPM metrics definitions will be subject to as rigorous a scrutiny for usefulness, clarity, and accuracy as other protocol standards. The IPPM WG will interact with other areas of IETF activity whose scope intersect with the requirement of these specific metrics. These include working groups such as BMWG, RMONMIB, and TEWG.

Goals and Milestones:
Done  Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done  Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done  Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done  Submit a document on connectivity to IESG for publication as an Informational RFC.
Done  Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
FEB 02  Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
FEB 02  Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
FEB 02  Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
FEB 02  Create initial draft on the definitions of link bandwidth capacity.
FEB 02  First draft for AS on one-way delay and loss.
MAR 02  Create initial draft on a sensitivity analysis of one-way delay and loss metric parameters (companion to the AS).
MAR 02  Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
MAR 02  Create initial draft on a MIB for reporting IPPM metrics.
APR 02  Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
APR 02  Create initial draft on comparing ITU and IETF IP performance metrics.
APR 02  Create initial draft on a packet reordering metric.
JUN 02  Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
JUL 02  Submit draft on bulk transfer capacity metric based on the bulk transfer framework (CAP) to the IESG for Proposed Standard.
OCT 02  Submit recommendation to the IESG on whether to advance, recycle, or deprecate RFCs 2678, 2679, 2680, and 2681 (connectivity, loss, & delay).
OCT 02  Submit draft on a packet reordering metric to the IESG for Proposed Standard.
DEC 02  Submit AS for one-way delay and loss to the IESG for PS.
DEC 02  Submit sensitivity analysis of one-way delay and loss, for consideration as an Informational RFC.
FEB 03  Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard.
MAR 03  Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
MAR 03  Discuss rechartering or ending working group.
Internet-Drafts:
  • - draft-ietf-ippm-ipdv-09.txt
  • - draft-ietf-ippm-loss-pattern-07.txt
  • - draft-ietf-ippm-npmps-07.txt
  • - draft-ietf-ippm-owdp-04.txt
  • - draft-ietf-ippm-owdp-reqs-02.txt
  • - draft-ietf-ippm-metrics-registry-01.txt
  • - draft-ietf-ippm-reordering-00.txt
  • - draft-ietf-ippm-reporting-mib-00.txt
  • - draft-ietf-ippm-owmetric-as-00.txt
  • Request For Comments:
    RFCStatusTitle
    RFC2330 I Framework for IP Performance Metrics
    RFC2678 E IPPM Metrics for Measuring Connectivity
    RFC2681 PS A Round-trip Delay Metric for IPPM
    RFC2680 PS A One-way Packet Loss Metric for IPPM
    RFC2679 PS A One-way Delay Metric for IPPM
    RFC3148 I A Framework for Defining Empirical Bulk Transfer Capacity Metrics

    Current Meeting Report

    Minutes of the IP Performance Metrics WG (ippm)
    Monday, July 15 at 13:00 - 15:00
    ====================================
    CHAIRS: Merike Kaeo <kaeo@merike.com>, Matt Zekauskas <matt@advanced.org>

    Al Morton took notes, which were edited into these minutes by the chairs.

    AGENDA:
    1. Agenda Bashing
    2. Working Group Milestone Status
    3. IPPM Reporting MIB and Metrics Registry Discussion
    4. Packet Reordering Discussion
    5. One-way Delay Protocol Requirements Discussion
    6. One-way Metrics Applicability Statement

    IETF home page: http://www.ietf.org/html.charters/ippm-charter.html
    IPPM home page: http://www.advanced.org/IPPM/

    3. IPPM Reporting MIB and Metrics Registry Discussion
    -- Emile Stephan
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-reporting-mib-00.txt
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-metrics-registry-00.txt

    Emile Stephan presented on the metrics registry and reporting MIB. First, the
    metrics registry document was discussed. The document defines a registry format,
    associating OIDs with IPPM metrics. RMON would like the registry so there are
    unique, common, ways of referencing the IPPM metrics. One issue yet to be
    resolved is how to define new metrics. This draft has a draft tree, where draft
    metrics can be placed to avoid ad-hoc assignment. The names would have to be
    changed when the metrics move into the "rfc" (standardized) branch. Emile is
    also placing metrics from IPPM drafts in the IESG last-call process into the
    standardized portion of the registry. Another issue raised is the metric OID
    length. He is in discussion with SNMP experts on this issue - one home node for
    the IPPM Metrics Registry could allow the metric object identifier to fit in
    exactly 8 bytes. There were no comments from the group on these issues.

    Next, Emile reported on the status of, and outstanding issues with, the
    reporting MIB. The first major issue is with timestamp resolution. His current
    proposal uses timestamps with 250 picosecond resolution. Is that enough for the
    next few years?

    Henk Uijterwaal made the comment that the proposed timestamp looks like it has
    the year 2100 problem - only two digits are displayed. He advocated using
    timestamps from NTP; then have multiple resolutions and all the tools exist for
    conversions. Jon Bennet supported this idea. Emile made a comment that you need
    something that could be used as an index, and you must have GMT not local time.
    You also need something human readable. Henk noted that NTP timestamps are GMT:
    tools exist for human-readable forms (also to convert from GMT to some local
    reference) and that as a 32 bit number it should be a suitable index. Emile will
    look into the issue, and there will be further discussion on the mailing list.

    Next, Emile noted that he added a field for synchronization (is clock
    synchronized or not?). A question from the floor asked about accuracy - Emile
    responded that the accuracy was in the MIB. Measurement management issues were
    raised. Is it useful to have history when one measure was more precise than
    other? Some seemed to agree that this was useful. Emile proposed SNMP over TCP
    as one way to help secure connections, so that you could control the
    measurements; he also made the point that it is important to have measurement
    packet interoperability. Matt Z. made statement that giving Emile feedback on
    control is fine, but it's not what we're doing in the working group. Al Morton
    noted that packet format is exactly what OWAMP is addressing.


    4. Discussion on Packet Reordering
    -- Al Morton
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-reordering-00.txt

    Al Morton presented status of current draft: it is a combination of the three
    drafts presented at the previous IETF. MLAS needs complete sequence and doesn't
    work if packets are lost, which is problematic, so it is not considered in this
    draft. (MLAS did have good property of identifying how many steps needed to
    restore order; it is just not clear how to apply it). The authors decided on two
    goals: first, determine whether packet order is maintained (and which packets
    are reordered). Second, quantify the extent of change.

    Al outlined the changes yet to be made, the principle one being the use of
    consistent notation. He also went over an extensive example showing the metrics
    in use (one comment was to make sure the examples in the draft show the
    application of all the metrics). Stanislav noted that the vector of arrival
    order in the example shown on the slides was not represented correctly - the
    notation should be fixed. Notation in the example section of the document should
    be checked too.

    Merike asked how many people read draft; about 15 people raised hands. There
    were no other substantive comments from the group. Merike asked people to take
    and further comments to mailing list!


    5. Discussion on One-Way Active Measurements Protocol Requirements -- Stanislav
    Shalunov
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-reqs-02.txt
    see also http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-04.txt

    Stanislav Shalunov presented the modifications to the OWAMP requirements
    document. In general, there were just textual modifications and small
    clarifications. Stanislav noted that the authors thought it was ready for
    publication. Matt took a rough consensus poll: approximately 5 had read the
    draft, and of those, all thought it should be moved forward. As a check Matt
    asked how many were interested; in the room approximately 10 raised their hands
    (most of the rest appeared to be lurkers). Al asked if the timestamp resolution
    comments from December were taken into account. Stanislav said that the
    timestamp resolution itself does not belong in requirement document. After
    discussion, Al was satisfied, but will reread the draft. Matt said the chairs
    would call for WG last call after the IETF.

    Stanislav then presented the changes in latest protocol draft, as well as
    pending changes. He asked for comments on pending changes before they actually
    go in the next version of the draft.

    A person from the audience (Sharam) asked a general question: why not use ICMP
    time stamps with a reliable return channel? You can then do everything from just
    the source. Stanislav pointed out that in abstract there was no problem with
    ICMP packets (as long as you have synchronized clocks), but you can't use them
    for accurate performance measurement. ICMP is not reliable, not treated like
    other packets in all networks, it may be identifiable, and it needs processing
    time at the responder (which may affect accuracy). With ICMP you can also not
    guarantee that the forward path is the same as the reverse path. The questioner
    mentioned that perhaps there could be some kind of traffic-engineered reverse
    path to "ensure" reliability.


    6. One-Way Metric Applicability Statement -- Henk Uijterwaal
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-owmetric-as-00.txt

    Finally, Henk Uijterwaal led a discussion of open issues with the one-way metric
    applicability statement draft. He started by noting that the draft should have a
    negative version number - it attempts to lay out the issues, but doesn't provide
    lots of answers. The intent is to use it as a basis for discussion; input is
    especially requested from other operators.

    The first issue was sending rate. Stanislav noted that the rate really depends
    on what you want to measure; looking for delay, one value is appropriate, for
    reordering you would want the rate to match a target application, and for
    capacity you want a much higher rate. He didn't see any single answer. (There
    was a "3% of link capacity discussion, which will be reported in a subsequent
    paragraph.) John Bennett noted that the rate needed might vary by time of day as
    well. Henk agreeded, but noted that bandwidth measurements certainly could swamp
    a link, although it might be appropriate if done infrequently.

    The next issue that generated discussion was packet size. Al Morton and
    Stanislav Shalunov were among the contributors. Again, the size depends on the
    purpose. IPDV requires one packet size, but that's only for a particular
    variation; there's no reason you can't calculate IPDV on different sized packet
    streams.

    Another issue was test duration. For delays, can do things continuously; for
    bandwidth you need to be careful and the test should be short-term. Al Morton
    noted that how frequently you report intermediate results determines how quickly
    you can act on it. He has a system that does this every 15 minutes, and that
    seems to be a good compromise. There might be some data loss in-between cycles,
    but it is a way to get fairly good coverage with a compromise.

    Another issue is data volume. 3% of link capacity was a figure from questioning
    providers. Note that it is a MAXIMUM over a LONG TERM. We are looking for
    providers that might violently disagree. You might also need to specify a
    distribution instead of just a blanket volume.

    The final issue that generated discussion for this document was security. Henk
    noted that there are potential DDoS problems: any single stream might be
    acceptable, but the total measurements might swamp a single receiver. (For
    situations when there are many senders to one receiver.) Stanislav mentioned
    that any kind of "amplifier" is not acceptable. For OWAMP, we're looking at
    authentication, and if a measurement point doesn't know you, then it will only
    send traffic back to you and not accept traffic from you.

    Finally, a group member asked about packet loss and errors. In the wireless
    area, very interested in loss due to errors instead of loss due to congestion.
    Was there a differentiation with IPPM? If you are looking at the network, you
    often don't have any observability into why a packet was lost. This question
    might be better addressed in tewg; Stanislav noted that the solution is sub-ip,
    so perhaps the measurement should be in sub-ip.

    Slides

    Agenda
    IPPM Reporting MIB & Registry
    Packet Reordering Metric for IPPM
    A One-way Active MEasurement Protocol
    One Way Metric Applicability Statement