2.8.5 IP Performance Metrics (ippm)

Last Modified: 2003-08-20

Chair(s):
Merike Kaeo <kaeo@merike.com>
Matthew Zekauskas <matt@internet2.edu>
Transport Area Director(s):
Allison Mankin <mankin@psg.com>
Jon Peterson <jon.peterson@neustar.biz>
Transport Area Advisor:
Allison Mankin <mankin@psg.com>
Technical Advisor(s):
Andy Bierman <abierman@cisco.com>
Mailing Lists:
General Discussion: ippm@ietf.org
To Subscribe: https://www1.ietf.org/mailman/listinfo/ippm
In Body: subscribe
Archive: http://www.ietf.org/mail-archive/working-groups/ippm/current/maillist.html
Description of Working Group:
Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied
to the quality, performance, and reliability of Internet data delivery
services. These metrics will be designed such that they can be
performed by network operators, end users, or independent testing
groups. It is important that the metrics not represent a value
judgment (i.e. define "good" and "bad"), but rather provide unbiased
quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as
NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and
procedures for accurately measuring and documenting these metrics.
The metrics are:

  - connectivity

  - one-way delay and loss

  - round-trip delay and loss

  - delay variation

  - loss patterns

  - packet reordering

  - bulk transport capacity

  - link bandwidth capacity

This is the cumulative set, including the metricsalready completed
and published.

The working group will closely review and then be guided by an
IESG document on how metrics advance along the standards track within
the IETF.  This document will also be relevant to the work of the
benchmarking working group (BMWG).  The first draft of this document
was discussed at IETF 51. Additionally, the WG will produce Proposed
Standard AS documents, comparable to applicability statements in RFC
2026, that will focus on procedures for measuring the individual
metrics and how these metrics characterize features that are important
to different service classes, such as bulk transport, periodic streams,
or multimedia streams. It is specifically out of scope for this working
group to actually characterize traffic, for example to characterize a
voice-over-IP stream. Each AS document will discuss the performance
characteristics that are pertinent to a specified service class;
clearly identify the set of metrics that aid in the description of
those characteristics; specify the methodologies required to collect
said metrics; and lastly, present the requirements for the common,
unambiguous reporting of testing results. Specific topics of these
AS documents must be approved by the Area Directors as charter
additions.

The WG will produce a protocol to enable communication among test
equipment that implements the one-way metrics. The intent is to create
a protocol that provides a base level of functionality that will allow
different manufacturer's equipment that implements the metrics
according to a standard to interoperate. A protocol requirements
document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM
metrics, such as one-way delay and loss, to facilitate the
communication of metrics to existing network management systems. Thus,
the group will create a MIB that contains predominantly read only
variables.  If, after the protocol requirements document is finished,
the group decides that it is appropriate to add variables that control
the underlying measurements that the metrics report, such a control
structure may be added as a separate document, subject to review by
the IESG.

The intent of the WG is to cooperate with other appropriate standards
bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote
consistent approaches and metrics. Within the IETF process, IPPM
metrics definitions will be subject to as rigorous a scrutiny for
usefulness, clarity, and accuracy as other protocol standards. The
IPPM WG will interact with other areas of IETF activity whose scope
intersect with the requirement of these specific metrics. These
include working groups such as BMWG, RMONMIB, and TEWG.
Goals and Milestones:
Done  Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done  Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done  Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done  Submit a document on connectivity to IESG for publication as an Informational RFC.
Done  Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
Done  Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
Done  Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
Done  Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
Feb 02  Create initial draft on the definitions of link bandwidth capacity.
Done  First draft for AS on one-way delay and loss.
Mar 02  Create initial draft on a sensitivity analysis of one-way delay and loss metric parameters (companion to the AS).
Done  Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
Done  Create initial draft on a MIB for reporting IPPM metrics.
Apr 02  Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
Apr 02  Create initial draft on comparing ITU and IETF IP performance metrics.
Done  Create initial draft on a packet reordering metric.
Jun 02  Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
Jul 02  Submit draft on bulk transfer capacity metric based on the bulk transfer framework (CAP) to the IESG for Proposed Standard.
Oct 02  Submit recommendation to the IESG on whether to advance, recycle, or deprecate RFCs 2678, 2679, 2680, and 2681 (connectivity, loss, & delay).
Oct 02  Submit draft on a packet reordering metric to the IESG for Proposed Standard.
Dec 02  Submit AS for one-way delay and loss to the IESG for PS.
Dec 02  Submit sensitivity analysis of one-way delay and loss, for consideration as an Informational RFC.
Feb 03  Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard.
Mar 03  Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
Mar 03  Discuss rechartering or ending working group.
Internet-Drafts:
  • - draft-ietf-ippm-owdp-06.txt
  • - draft-ietf-ippm-owdp-reqs-06.txt
  • - draft-ietf-ippm-metrics-registry-04.txt
  • - draft-ietf-ippm-reporting-mib-03.txt
  • - draft-ietf-ippm-reordering-03.txt
  • - draft-ietf-ippm-owmetric-as-01.txt
  • Request For Comments:
    Framework for IP Performance Metrics (RFC 2330) (94387 bytes)
    IPPM Metrics for Measuring Connectivity (RFC 2678) (18087 bytes)
    A One-way Delay Metric for IPPM (RFC 2679) (43542 bytes)
    A One-way Packet Loss Metric for IPPM (RFC 2680) (32266 bytes)
    A Round-trip Delay Metric for IPPM (RFC 2681) (44357 bytes)
    A Framework for Defining Empirical Bulk Transfer Capacity Metrics (RFC 3148) (36041 bytes)
    One-way Loss Pattern Sample Metrics (RFC 3357) (30570 bytes)
    IP Packet Delay Variation Metric for IPPM (RFC 3393) (47731 bytes)
    Network performance measurement for periodic streams (RFC 3432) (52493 bytes)

    Current Meeting Report

    IETF IP Performance Metrics WG (ippm) Tuesday, July 15, 2003 at 09:00 to 
    11:30 
    ========================================
    
    The meeting was moderated by the working group chairs, Matt Zekauskas and 
    Merike Kaeo.  Al Morton, and Henk Uijterwaal took notes, which were 
    edited into these minutes by the chairs.
    
    AGENDA 
    1. Agenda Bashing 
    2. Packet Reordering 
    3. Reordering Density 
    4. IPPM-MIB 
    5. Status, Milestones & Futures
    
    
     2. Packet Reordering
        --Al Morton
    
    Al Morton led off the meat of the meeting with a report on the 
    reordering metric progress.  Many editorial changes were made from -02 to 
    -03 to clarify text, and there are more in progress.  A 
    reordering-free-run metric  was added based on Jon Bennett's comments in the 
    last meeting.  This metric characterizes how often reordering happens 
    (along with previous gap metric and general frequency metric).
    
    Comment from Jon: want to answer the question how often do you get 
    packets out of order.  This is important for some applications that need 
    in-order packets.  Applications that run well when in-order (or 
    increasing order) want a feel for how much "order" there is. Matt asked why 
    is gap metric not good enough. Jon replied that it doesn't show how often 
    reordering happens.
    
    Basically, the 'beginning of reordering event' is defined.  You can't know 
    when the event ends, in order to keep the metric orthogonal to loss. Call 
    the places where a sequence number increases, but skips the  
    'reordering discontinuity'.  The Reordering Gap is the difference 
    between discontinuities.  The reordering free run says how many packets 
    arrive in order.  To some extent, the gap shows the "early packets", while 
    reordering free shows the "late packets".
    
    Al mentioned two pending updates.  First, separating reordering by packet 
    sequence numbers and other views (time, byte counts (TCP sequence 
    numbers). Second, clarify the base "nonreversing sequence" metric so that it 
    reports "true" for packets that are out of sequence by it's 
    definition, false otherwise (currently, it's not as clear and 
    inverted).
    
    One open issue that was mentioned: dealing with fragmentation.  The 
    current draft says that you only consider reassembled IP datagrams (and 
    that's all it says).  Jerry Perser (spirent) would like all 
    reordering to be noted -- reassembly could hide the case where a 
    datagram was fragmented, the fragments reordered, and then 
    reassembled.  Jerry will supply some text for the next draft revision. 
    
    
    
     3. Reordering Density
        --Jerry McCollom
    
    Next, Jerry McCollom from Agilent gave a presentation on reordering 
    density, a separate characterization metric developed at the 
    University of Colorado at Fort Collins.  The authors were not present, but 
    Jerry has been working with them, and presented slides they produced (see 
    slides).  The metric has a model of a buffer, and looks at how many 
    packets are stored in the buffer to compute the metric. If the buffer 
    overflows, those packets are considered lost.  If the duplicates arrive, 
    they are ignored.  The audience (and mailing list members) have 
    questioned this mixing of loss and reordering -- there will be a new draft 
    after the meeting that should address (some of?) the concerns.
    
    This led to a good general reordering discussion, including adding usage 
    notes to each metric (where it works well, when it fails) and having each 
    metric indicate when it is out-of-scope (if such a condition exists).
    
    Greg Ryan made a number of points, including that it is "squishy" as to 
    what exactly is measured; the drafts should explain what the metric 
    measures compared to a complete characterization of reordering (perhaps 
    edit distance), and give some examples.  It is very important to specify 
    when the "domain of validity" is exceeded... for example, the 
    underlying assumption with both drafts is that reordering does not happen 
    very often (otherwise "frequency of reordering" and "reordering free gaps" 
    don't make much sense).  What happens if the data is "completely messed up" 
    -- say the packets arrive in truly random order. Greg is interested in 
    providing some text to Al about edit distance. He will comment to the 
    list.
    
    Jon Bennett probed how reordering density interacts with packet loss. 
    Jerry said that the metric is density not loss.  The metric assumes a 
    threshold (the buffer size) where packet is lost and leaves it at that. The 
    authors believe that a characterization of loss needs to be added; how 
    often are packets lost?
    
    Jon asked what happens if the threshold is wrong?  It depends on the 
    application data stream in the end.  So Jon wondered if the metric tried to 
    represent a prototypical application... and then time might become more 
    relevant than distance, especially for real time applications.  Jerry said 
    that without much thought it seems that it's possible to compute a 
    sensible threshold that would cover a large number of 
    applications. Jon noted that the metric keeps state - the packets in he 
    buffer.  Thus it seems like there might be a number of metrics for 
    different applications.  Either a metric tells me something 
    intuitively or it's specific to an application.  It seems like this 
    metric is too complicated to be intuitive, but not complicated enough to 
    match an application.
    
    Jon also noted that with "tool devices", having to keep state 
    (particularly on a high-speed test) is impossible, or places a high load on 
    the device.  You have to accept that there may be some 
    deficiency... but being loss impervious might be more practical than 
    trying to emulate an application.
    
    Al thanked Jerry for coming to represent the metric. It's difficult to 
    maintain orthogonality between loss and reordering, but that's what the 
    current reordering draft tries to do.  In the -00 draft of reordering 
    density, there is an example where the metric counts "early packets" out of 
    order.  The current draft calls "late packets" out of order, for one thing to 
    distinguish loss (only way to distinguish is to have the late packet 
    arrive).
    
    Jerry M.  mentioned that in the second examples... some of packets in 
    buffer reordered, doesn't catch.  Before releasing the packets, one could 
    take a look at what have in buffer to accurately reflect metric.
    
    Al thought that sounded reasonable... he noted that one of the problems in 
    the industry right now is that multiple vendors look at the same same 
    stream would call different packets out of order; we need one 
    definition that everyone agrees to.
    
    Jerry Purser wanted to pursue how easily such a metric could be 
    interpreted, say by a technical support person. Look at the first graph on 
    slide 9.  What does it tell you?  It looks like two packets got pushed out.  
    Al noted that two packets out of order, but the chart has three bars. 
    Jerry M. noted that the 0 slot represents in-order packets.
    
    Jerry P. was trying to understand normalization...   How many times was 
    packet 3 displaced?  That's not what we're measuring. Merike noted it was 
    how many times packet was in a particular buffer slot.
    
    Jerry M said that if packet 3 is what caused us to start buffering in the 
    first place... one tweak could be to see 3 and release, 
    displacement of 3 for packet.
    
    Jerry P was trying  to think as support person on line.  Get that chart, can 
    you figured out what happened from that chart? 55% of time everyting is in 
    order  45% of packets experienced some reordering Jerry P was 
    wondering if it was reordering or buffering.
    
    
    
    
     4. IPPM-MIB
        --Emile Stephan
    
    Next was Emile Stephan to report on Reporting MIB progress.  Major 
    changes include VACM support for security instead of 
    roll-your-own, and tweaking all the tables so that VACM access made sense 
    (mainly replacing pointers with data in some cases).  Emile also added 
    'burst' and 'multiburst' packet types -- MattZ asked where this came from, 
    and didn't get a clear answer.  He will take it to the mailing list.
    
    No comments from floor, other than Andy Bierman who noted there were still a 
    number of problems (probably due to the major reorganization to satisfy 
    VACM).  The chairs asked had anyone read draft.  No responses other than 
    Andy in this audience.  The chairs then followed up with who would use it.  
    No responses in this audience.  This is different from the response in 
    earlier meetings, where there was general interest.
    
    
    
     5. Status, Milestones & Futures
        --Matt Zekauskas
    
    Finally, Matt once again went over the milestones.  The OWAMP 
    requirements document was cycling between the author and the IESG for 
    clarification of the security section.  OWAMP itself was being updated 
    based on a sample implementation.  More information on the 
    implementation is available at 
    http://owamp.internet2.edu/ .  The MIB work has been progressing.  The 
    metrics registry MIB document has gone through last call and will be 
    submitted to the IESG.
    
    Matt noted areas where there was interest, but it had waned.  (In 
    particular: Cap, a BTC implementation; parameter sensitivity; ITU vs IPPM 
    metrics; path bottleneck definitions; and the applicability 
    statement.)  On the applicability statement: Merike and Henk 
    Uijterwaal have repeatedy prodded vendors, and they indicate interest and 
    promise text, but the text never materializes.  They want to shelve it as an 
    ippm item for now.  Perhaps interest might be generated in doing it in 
    'ispmon', should it become a WG from a BOF? MattZ noted that a probe will be 
    sent to the list, and if no interest is generated, the chairs would drop the 
    items from the charter milestones.
    
    Al noted that the advancing metrics draft was really needed.  He wanted to 
    know if the -00 draft was still available even though it had expired?   
    Matt said that he had one, and that it was also available from some of the 
    unofficial internet-drafts 

    Slides

    Agenda
    Packet Reordering Metric for IPPM
    Reorder Density (RD): Metric for Degree of Reordering in Packet Sequences
    IPPM REPORTING MIB
    IETF IPPM WG Status and Futures