2.8.5 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 56th IETF Meeting in San Francisco, California USA. It may now be out-of-date.

Last Modified: 2003-02-19

Chair(s):
Merike Kaeo <kaeo@merike.com>
Matthew Zekauskas <matt@internet2.edu>
Transport Area Director(s):
Scott Bradner <sob@harvard.edu>
Allison Mankin <mankin@psg.com>
Transport Area Advisor:
Allison Mankin <mankin@psg.com>
Technical Advisor(s):
Andy Bierman <abierman@cisco.com>
Mailing Lists:
General Discussion: ippm@ietf.org
To Subscribe: request-ippm@ietf.org
In Body: subscribe
Archive: http://www.ietf.org/mail-archive/working-groups/ippm/current/maillist.html
Description of Working Group:
Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgment (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and procedures for accurately measuring and documenting these metrics. The metrics are:

- connectivity

- one-way delay and loss

- round-trip delay and loss

- delay variation

- loss patterns

- packet reordering

- bulk transport capacity

- link bandwidth capacity

This is the cumulative set, including the metricsalready completed and published.

The working group will closely review and then be guided by an IESG document on how metrics advance along the standards track within the IETF. This document will also be relevant to the work of the benchmarking working group (BMWG). The first draft of this document was discussed at IETF 51. Additionally, the WG will produce Proposed Standard AS documents, comparable to applicability statements in RFC 2026, that will focus on procedures for measuring the individual metrics and how these metrics characterize features that are important to different service classes, such as bulk transport, periodic streams, or multimedia streams. It is specifically out of scope for this working group to actually characterize traffic, for example to characterize a voice-over-IP stream. Each AS document will discuss the performance characteristics that are pertinent to a specified service class; clearly identify the set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of testing results. Specific topics of these AS documents must be approved by the Area Directors as charter additions.

The WG will produce a protocol to enable communication among test equipment that implements the one-way metrics. The intent is to create a protocol that provides a base level of functionality that will allow different manufacturer's equipment that implements the metrics according to a standard to interoperate. A protocol requirements document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM metrics, such as one-way delay and loss, to facilitate the communication of metrics to existing network management systems. Thus, the group will create a MIB that contains predominantly read only variables. If, after the protocol requirements document is finished, the group decides that it is appropriate to add variables that control the underlying measurements that the metrics report, such a control structure may be added as a separate document, subject to review by the IESG.

The intent of the WG is to cooperate with other appropriate standards bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote consistent approaches and metrics. Within the IETF process, IPPM metrics definitions will be subject to as rigorous a scrutiny for usefulness, clarity, and accuracy as other protocol standards. The IPPM WG will interact with other areas of IETF activity whose scope intersect with the requirement of these specific metrics. These include working groups such as BMWG, RMONMIB, and TEWG.

Goals and Milestones:
Done  Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done  Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done  Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done  Submit a document on connectivity to IESG for publication as an Informational RFC.
Done  Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
FEB 02  Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
FEB 02  Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
FEB 02  Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
FEB 02  Create initial draft on the definitions of link bandwidth capacity.
FEB 02  First draft for AS on one-way delay and loss.
MAR 02  Create initial draft on a sensitivity analysis of one-way delay and loss metric parameters (companion to the AS).
MAR 02  Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
MAR 02  Create initial draft on a MIB for reporting IPPM metrics.
APR 02  Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
APR 02  Create initial draft on comparing ITU and IETF IP performance metrics.
APR 02  Create initial draft on a packet reordering metric.
JUN 02  Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
JUL 02  Submit draft on bulk transfer capacity metric based on the bulk transfer framework (CAP) to the IESG for Proposed Standard.
OCT 02  Submit recommendation to the IESG on whether to advance, recycle, or deprecate RFCs 2678, 2679, 2680, and 2681 (connectivity, loss, & delay).
OCT 02  Submit draft on a packet reordering metric to the IESG for Proposed Standard.
DEC 02  Submit AS for one-way delay and loss to the IESG for PS.
DEC 02  Submit sensitivity analysis of one-way delay and loss, for consideration as an Informational RFC.
FEB 03  Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard.
MAR 03  Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
MAR 03  Discuss rechartering or ending working group.
Internet-Drafts:
  • - draft-ietf-ippm-owdp-05.txt
  • - draft-ietf-ippm-owdp-reqs-05.txt
  • - draft-ietf-ippm-metrics-registry-02.txt
  • - draft-ietf-ippm-reporting-mib-02.txt
  • - draft-ietf-ippm-reordering-02.txt
  • - draft-ietf-ippm-owmetric-as-01.txt
  • Request For Comments:
    RFCStatusTitle
    RFC2330 I Framework for IP Performance Metrics
    RFC2678 E IPPM Metrics for Measuring Connectivity
    RFC2679 PS A One-way Delay Metric for IPPM
    RFC2680 PS A One-way Packet Loss Metric for IPPM
    RFC2681 PS A Round-trip Delay Metric for IPPM
    RFC3148 I A Framework for Defining Empirical Bulk Transfer Capacity Metrics
    RFC3357 I One-way Loss Pattern Sample Metrics
    RFC3393 PS IP Packet Delay Variation Metric for IPPM
    RFC3432 PS Network performance measurement for periodic streams

    Current Meeting Report

    IP Performance Metrics WG (ippm)
    Tuesday, March 18, 2003 at 15:45 to 16:45
    
    =========================================
    
    The meeting was moderated by the working group chairs, Matt Zekauskas and 
    Merike Kaeo.  Dave McDysan took notes, which were edited into these 
    minutes by the chairs.  Thanks also to Simon Leinen, who scribed the 
    meeting into Jabber; some of his notes were used while generating this 
    text.
    
    
    AGENDA:
    1. Agenda bashing, Working Group Milestones Status
    2. One-way Metric Applicability Statement
    3. Discussion on One-Way Active Measurements Protocol
    4. Discussion on Packet Reordering
    5. IPPM Reporting MIB
    6. IPPM Metrics Registry
    
    
    1. Agenda bashing & Working Group Milestones Status
    
    There were no suggestions to modify the agenda.
    
    Slides presented by Matt Z noted a new RFC, and overviewed the current 
    work.  Matt called for Implementation Reports to move to existing 
    metrics to draft standard - send Email to a chair if you know of an 
    implementation.  (Advancement requires an advancing metrics RFC 
    approved by the transport ADs and the IESG first, however; that work is 
    currently in the Transport WG.)
    
    Matt noted there were a number of milestones that had no progress.  If no 
    interest is expressed, they will be dropped from the charter.  We are 
    looking for people to contribute on the following topics (assuming the 
    group still feels they are worthwhile):
    
      * parameter sensitivity
      * ITU (e.g., Y.1540) vs IETF metrics
          Al Morton indicated that these are similar, recommend dropping this 
    item from the charter
      * Path bottleneck definitions,
      * CAP (or other BTC metric)
    
    
    
    2. One-way Metric Applicability Statement
      --Henk Uijterwaal
    
    Henk noted that there were no comments on the draft since the Atlanta 
    meeting, and that perhaps it just wasn't time to do this work.  
    However, Merike has been working to see if there was interest in the 
    provider community, and today Henk and Merike met with people from two 
    providers and they were interested in moving the work forward.  They were 
    looking for specific questions; Henk and Merike will identify areas where 
    input is needed from people, and they will send questions off to 
    providers.  They would welcome help.
    
    
    
    3.  Discussion on One-Way Active Measurements Protocol
       -- Stanislav Shalunov
    
    First, the requirements document.
    
    
    http://www.ietf.org/internet-drafts/draf
    t-ietf-ippm-owdp-reqs-06.txt
    
    The requirements document passed WG last call, and IESG comments are being 
    addressed.  Steve Bellovin was worried about replay attacks; but in some 
    sense that is what we want to measure... we would like to measure 
    duplication, which may or may not be malicious.  Similarly, an attacker 
    could add delay and we would like to measure that as well.  We do have a 
    requirement that traffic is hard to spot, for example there should be no 
    fixed port number.  It would be non-trivial to place OWDP traffic in a 
    separate queue.
    
    Allison Mankin came to the mike as AD.  She mentioned that 
    authorization text for control protocol design needs to be 
    strengthened - it needs to be more than IP address per current draft.  
    OWAMP needs strong authentication for users since it is a powerful 
    protocol. Anonymous identity wording was also problematic. Stanislav 
    clarified that cryptographic authentication implementation is 
    required; however, a user could request a less secure 
    authentication (e.g., based on IP addresss) Allison, Stanislav, Merike to 
    agree on wording -- could be along lines of clarifying mandatory 
    implementation but optional invocation in the security 
    considerations section.
    
    Next, we moved to the current OWAMP protocol specification.
    
    http://www.ietf.org/internet-drafts/draf
    t-ietf-ippm-owdp-06.txt
    
    There are changes in the current draft based upon implementation 
    experience, they are shown in the slides.  Three new items: First, 
    advertise server uptime at the beginning of each connection (helps with 
    long-term monitoring) -- this disambiguates between whether the other end 
    crashed or the network failed.  Second, stop-sessions now supplies number of 
    packets sent in each appropriate session to help decide if the last 
    packet was lost.  Third, the algorithm to compute exponentially 
    distributed pseudo-random deviates was documented, and sample code 
    supplied to ensure that all implementations generate the same 
    sequence.  Vectors will be added to allow code testing.
    
    
    
    4. Discussion on Packet Reordering
      -- Al Morton
    
    http://www.ietf.org/internet-drafts/draf
    t-ietf-ippm-reordering-02.txt
    
    As background, group asked to review Stanislav's n-reordering draft.  
    
    http://www.ietf.org/internet-drafts/draf
    t-shalunov-reordering-definition-02.txt
    
    There was an independent draft from Colorado State also submitted, but none 
    of the authors were at meeting.  A minority of people in the room had read 
    the draft.  Al gave a 'tutorial' on reordering.  He noted that the 
    independent draft -- a "density" metric -- took into account lost 
    packets, and was therefore not acceptable.  Out of order packets can be 
    viewed as "early" or "late".  "Late" packets trigger reordering 
    computation at the receiver.
    
    We had trouble merging all metrics to section 4.2, in particular we could 
    not include Stanislav's n-reordering without change.  However, there was 
    significant editing to unify notation.  A "gap" metric was added 
    (section 4.4) based on a input from Jon Bennett, as the distance between the 
    beginning of reordering events.  Matt Mathis asked how this handle nested 
    reordering events caused by multiple reordering events. e.g., the 
    arrival is 1,2,4,5,6,7,9,10,11,8,3,12.  Jon stated that this metric might be 
    trying to keep track of too much state and structure. Authors to think 
    about this and discuss.
    
    Al proposed to fix the problem (with merging the metrics) in the draft by 
    reorganization.  Keep section 3 (determining whether or not packet order is 
    maintained), then to quantify the extent of the changes split into two 
    sections:
    
     Sec 4 - Metrics leaning to network characterization (frequency, 
    distance, and a metric for multiple events (e.g., reordering gap).
    
     Sec 5 - Metrics primarily for receiver assessment.  n-reordering goes here 
    (n=3 predicts NewReno TCP unnecessary retransmissions).  Potentially a 
    modified version of the reordering density metric proposed.   Matt noted 
    that we should be discerning when adding metrics; there should be a good 
    reason for adding another.
    
    A question from the group asked if reordering distance referred to a 
    distribution or average.  Al noted that it depends on whether you are 
    talking about singleton metrics or statistics.  The questioner 
    clarified that he was assuming there are multiple samples.  Stanslav 
    mentioned mentioned that you can have a complete distribution if the 
    number of values presented is small, and can then generate 
    statistics.
    
    Al presented his idea of a potential fix to make n-reordering match the 
    4.2.3 singleton definition is to change "all" to "ANY" in Definition 1.  The 
    intent is to identify all packets that participate in a reordering event.  
    [Chair note: We believe Al is looking to change the statement from 
    universally quantified to existentially quantified.  The proposed fix 
    conveys this idea, but isn't correct mathematically (it has the same 
    semantics as the original statement).]
    
    Greg Woodhouse asked if the measure is trying to quantify amount of 
    effort (resources steps, memory) to restore order?  Is it a hamming 
    distance?  Al Morton replied no, it is trying to quantify amount of 
    buffer needed to restore order.  Many metrics do not consider loss. 
    Stanislav said that n-reordering was measuring the quantity of packets that 
    are as good as lost for natural application behavior.
    
    Dave McDysan asked if the VoIP example dropped from Stanislav's draft?  
    Stanislav answered that it was not intended to be dropped, will look at 
    this again.  Al Morton noted that what is important for VoIP is the time 
    measure of reordering (a packet that arrives too late is as good as 
    lost).  In addition, VoIP jitter absorption buffers can reset.
    
    
    
    5. IPPM Reporting MIB
      -- Jessie Jewitt
    
    http://www.ietf.org/internet-drafts/draf
    t-ietf-ippm-reporting-mib-02.txt
    
    See slides for detailed changes.
    
    Lots of editing for simplification, presentation, and to correct errors
    found when running SMILINT.  They have a working prototype SNMP agent.
    
    TypeP was changed from an octet string to snmpadminstring for 
    readability.
    
    Added scalars to control what happens when a log is full; suspend, resume 
    and wrap the log were defined based on previous CMIP experience.  
    Comments are solicited.
    
    A number of changes were made to the tables, based on 
    implementation and working group suggestions.  See slides.
    
    A couple of open issues were noted.  First, is there a need for a count 
    parameter for TypeP now that it is text instead of binary?  Second, is 
    there a problem with spaces in TypePaddress?  Perhaps it too needs a count 
    parameter.
    
    There were no immediate comments on these issues.
    
    Andy Bierman had some comments on the MIB, which he will send in detail to 
    the list.  In the meeting, Andy noted that some of the changes were not 
    consistent across ippm OwnerString, SnmpAdminString.  There is also still 
    overlap with the control provided by the owners and shared-owners table and 
    VACM. [A standard SNMPv3 access control method.]  Emile noted that in the 
    module compliance section this control is not mandatory if VACM is used; 
    Andy did not feel that would be enough to get approved.  The 
    recommendation is to remove this control.
    
    Andy noted that the MIB has been cleaned up substantially; and that this is a 
    very heavyweight (from an implementation resource point of view) MIB.  This 
    is in the same category as RMON probes with a dedicated processor and 
    significant storage.
    
    The limit of the index on history (64K entries) seemed artificially low; 
    Andy suggested that some case study thinking should be done to see how 
    quickly this would wrap.  Why wasn't it just made a 32 bit quantity?
    
    
    
    6. Metrics Registry discussion
      --Emile Stephan
    
    
    http://www.ietf.org/internet-drafts/draf
    t-ietf-ippm-metrics-registry-02.txt
    
    Finally, and with only a few minutes remaining, Emile Stephan 
    presented the changes to the metrics registry.  See the slides for 
    specific points.  The biggest changes are with the tree (the "draft" 
    subtree was removed), and instructions for draft authors that want to add a 
    new metric to the registry were clarified.
    
    Emile thinks that the draft is stable, and is looking for a last call.  He 
    suggested that people review by mid-April, and then a last call should be 
    issued.
    
    Andy Bierman noted that the registry looks good; it has been cleaned up.  He 
    also stated that RMON needs the registry.
    
    Matt said that a last-call will begin soon, and reiterated that the list 
    will be moved to ietf.org (ippm@ietf.org) soon after this meeting.
    

    Slides

    Agenda
    A One-way Active Measurement Protocol - Requirements
    Packet Reordering Metric for IPPM
    IPPM Reporting MIB
    IPPM Metrics Registry