2.8.7 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 61st IETF Meeting in Washington, DC USA. It may now be out-of-date.

Last Modified: 2004-10-15

Chair(s):

Henk Uijterwaal <henk@ripe.net>
Matthew Zekauskas <matt@internet2.edu>

Transport Area Director(s):

Allison Mankin <mankin@psg.com>
Jon Peterson <jon.peterson@neustar.biz>

Transport Area Advisor:

Allison Mankin <mankin@psg.com>

Technical Advisor(s):

Andy Bierman <abierman@cisco.com>

Mailing Lists:

General Discussion: ippm@ietf.org
To Subscribe: https://www1.ietf.org/mailman/listinfo/ippm
In Body: subscribe
Archive: http://www.ietf.org/mail-archive/web/ippm/index.html

Description of Working Group:

Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied
to the quality, performance, and reliability of Internet data delivery
services. These metrics will be designed such that they can be
performed by network operators, end users, or independent testing
groups. It is important that the metrics not represent a value
judgment (i.e. define "good" and "bad"), but rather provide unbiased
quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as
NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and
procedures for accurately measuring and documenting these metrics.
The metrics are:

  - connectivity

  - one-way delay and loss

  - round-trip delay and loss

  - delay variation

  - loss patterns

  - packet reordering

  - bulk transport capacity

  - link bandwidth capacity

This is the cumulative set, including the metricsalready completed
and published.

The working group will closely review and then be guided by an
IESG document on how metrics advance along the standards track within
the IETF.  This document will also be relevant to the work of the
benchmarking working group (BMWG).  The first draft of this document
was discussed at IETF 51. Additionally, the WG will produce Proposed
Standard AS documents, comparable to applicability statements in RFC
2026, that will focus on procedures for measuring the individual
metrics and how these metrics characterize features that are important
to different service classes, such as bulk transport, periodic streams,
or multimedia streams. It is specifically out of scope for this working
group to actually characterize traffic, for example to characterize a
voice-over-IP stream. Each AS document will discuss the performance
characteristics that are pertinent to a specified service class;
clearly identify the set of metrics that aid in the description of
those characteristics; specify the methodologies required to collect
said metrics; and lastly, present the requirements for the common,
unambiguous reporting of testing results. Specific topics of these
AS documents must be approved by the Area Directors as charter
additions.

The WG will produce a protocol to enable communication among test
equipment that implements the one-way metrics. The intent is to create
a protocol that provides a base level of functionality that will allow
different manufacturer's equipment that implements the metrics
according to a standard to interoperate. A protocol requirements
document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM
metrics, such as one-way delay and loss, to facilitate the
communication of metrics to existing network management systems. Thus,
the group will create a MIB that contains predominantly read only
variables.  If, after the protocol requirements document is finished,
the group decides that it is appropriate to add variables that control
the underlying measurements that the metrics report, such a control
structure may be added as a separate document, subject to review by
the IESG.

The intent of the WG is to cooperate with other appropriate standards
bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote
consistent approaches and metrics. Within the IETF process, IPPM
metrics definitions will be subject to as rigorous a scrutiny for
usefulness, clarity, and accuracy as other protocol standards. The
IPPM WG will interact with other areas of IETF activity whose scope
intersect with the requirement of these specific metrics. These
include working groups such as BMWG, RMONMIB, and TEWG.

Goals and Milestones:

Done  Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done  Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done  Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done  Submit a document on connectivity to IESG for publication as an Informational RFC.
Done  Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
Done  Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
Done  Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
Done  Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
Done  First draft for AS on one-way delay and loss.
Done  Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
Done  Create initial draft on a MIB for reporting IPPM metrics.
Done  Create initial draft on a packet reordering metric.
Aug 04  Create initial draft on the definitions of link bandwidth capacity.
Done  Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
Aug 04  Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
Oct 04  Submit draft on a packet reordering metric to the IESG for Proposed Standard.
Dec 04  Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard.
Dec 04  Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
Dec 04  Collect implementation reports for RFCs 2678-2681
Feb 05  Discuss rechartering or ending working group.

Internet-Drafts:

  • draft-ietf-ippm-owdp-11.txt
  • draft-ietf-ippm-metrics-registry-07.txt
  • draft-ietf-ippm-reporting-mib-06.txt
  • draft-ietf-ippm-reordering-07.txt

    Request For Comments:

    RFCStatusTitle
    RFC2330 I Framework for IP Performance Metrics
    RFC2678 E IPPM Metrics for Measuring Connectivity
    RFC2679 PS A One-way Delay Metric for IPPM
    RFC2680 PS A One-way Packet Loss Metric for IPPM
    RFC2681 PS A Round-trip Delay Metric for IPPM
    RFC3148 I A Framework for Defining Empirical Bulk Transfer Capacity Metrics
    RFC3357 I One-way Loss Pattern Sample Metrics
    RFC3393 PS IP Packet Delay Variation Metric for IPPM
    RFC3432 PS Network performance measurement for periodic streams
    RFC3763 I A One-way Active Measurement Protocol Requirements

    Current Meeting Report



    IP Performance Metrics WG (ippm)
    Tuesday, November 9 at 0900-1130
    =================================

    The meeting was chaired by Henk Uijterwaal and Matt Zekauskas. Al Morton acted as scribe with additional notes by Matt; raw notes were edited into these minutes by the chairs.

    Agenda:

    1. Review of Agenda and Status of Milestones
    2. One-Way Active Measurements Protocol
    3. Reordering drafts
    4. Implementation reports for RFC2678-2681.
    5. Multiparty Communications Parameters and Metrics.
    6. Status of the Link Bandwidth Draft.
    7. Discussion on an open network performance testing protocol.



    1. Review of Agenda and Status of Drafts and Milestones

    Henk opened the meeting with a review of the agenda. No one had any suggestions for further changes. He then went on to review working group status. The link bandwidth draft was due in August; although there is no draft to date there has been progress, and Phil Chimento will report on current thinking later. There are four drafts in progress. The MIB draft did not show sufficient interest on the mailing list, so it will be dropped from the charter; it can, however, go on as an individual draft so the work done to date is not lost. Two new work items have been proposed: multi-to-multi measurements and a way to store traceroutes. A personal draft on multiparty communication parameters and metrics has been created and announced to the list; no draft has been created for traceroutes.


    2. One-Way Active Measurements Protocol
    -- Stanislav Shalunov
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-11.txt

    A summary of all the changes is in the slides. There was a substantial wire-format change to robustly differentiate between packets that were sent and not received and packets that were never sent (if, for example, a measurement host had internal sending delays with a relatively small timeout, such that packets would have been sent too late to be received. The ability to start in the past was removed (as the packets are just not sent anyway); there is a new section on timing of packets as sent that clarifies and makes explicit some implications that existed in the old text. The security section has been expanded substantially to clarify requirements and explain design decisions. Return codes in the "ACCEPT field" have been expanded; there was some discussion of what it means to be an internal error versus the catchall failure; internal errors are completely unexpected - something wrong in the underlying implementation. There is now a MUST to remember the list of skipped packets (which can happen for a number of practical reasons).

    Stanislav ended by noting that we still need a security review; Steve Bellovin asked two people to review; one could not, and the other has not yet completed his review. Jeff Dunn from SI international offered to provide a review, and the chairs will push on the IESG folks again.



    3. Reordering drafts
    -- Al Morton
    http://www.ietf.org/internet-drafts/draft-ietf-ippm-reordering-07.txt

    Al Morton reported on the status of the reordering draft. There were a number of substantive comments from last call on the mailing list. See slides for the proposed changes based on the last call comments.

    One issue that Al brought up that requires closure is that of duplicate packets. Handling duplicate packets has memory implications that vary by implementation. Al did not feel that we could just ignore duplicate packets. The current draft has suggestions on dealing with the memory implications - moreso than in previous IPPM metrics (or in BMWG). These other drafts have been accepted, why is the memory implication now a large issue? Al proposes that if no further comments are received, the extra documentation closes this issue. Randy Presuhn noted (prior to meeting) that we should document the closure of this issue explicitly. There were no objections during the meeting.

    Al will revise the draft, and we will last call the draft again. There were no comments on the proposed changes from the floor. Al asked if he could get an additional person to read and comment on the draft; Phil Chimento offered to do so.



    4. Implementation reports for RFC2678-2681.

    Henk once again called for implementation reports. We've received three so far. Keynam Hedayat from Brix offered an implementation report; Al Morton said he would try to do one as well. Henk asked that any reports be received by mid-December; this would allow the chairs to compile them over the holidays.



    5. Multiparty Communications Parameters and Metrics.
    draft-liang-multiparty-para-00.txt

    The author has asked us to discuss the draft and see if it should be picked up as a working group item. This is a personal draft that was announced on the mailing list, but there has been no comment so far. Roughly, this draft concerns the QoS requirements of multiparty communication services, and places them in terms of measurement parameters. It tries to derive a set of parameter metrics from the existing one-way metrics in IPPM. Henk asked if anyone in the audience had read the draft, no one had. Therefore the chairs did not ask if it should be a working group item. Matt noted that this draft is reminiscent of Emile Stephan's spatial metrics draft. Emile said he would read and comment.




    6. Status of the Link Bandwidth Draft.
    --Phil Chimento

    Mark Allman pointed Phil in the direction of a workshop held at CAIDA last December, and he started with that to take a survey of the current tools and techniques. (See the slides.) There were many tools with three different base techniques, and these techniques (and tools) all measure different things. Phil listed a bunch of difficulties in creating a metric.

    Phil asked what we should focus on in the draft. We could do a critical analysis of current tools; provide definitions; start at first principles and create definitions, required metrics, and statistical analyses.

    Matt mentioned that the original intent was to simply attack the definitions. Jeff Dunn supported a rigorous approach; note requirements, which techniques were good, which were questionable, and a statistical analysis. He felt that there was questionable reporting and inferences drawn in what he sees; and he would like to see "solid math" and calculus of probability applied to this area to see what is going on. He also noted that MPLS is another thing that should be added to Phil's difficulty list -- it could add overhead.

    Matt Mathis noted that there was a general problem with this class of metrics, because operators simply measure utilization with SNMP. He asked if any ISPs are doing this... or would it just be used to check an ISP's service. He also noted that SLA generation based on metrics was not an area the IESG deemed the group should pursue. Cynthia Martin said the DoD uses this to assure they get the bandwidth contracted for.

    Joeseph Issac said the draft should have definitions and overview of tools, but detailed statistics is probably too much given the state of the art. If we want a document describing statistical analysis, it should be separate. Someone that worked for MCI said that this was a good proposal; we should have definitions and not approach the methodology. What matters is the time measurements are taken, and traffic patterns, these considerations should be documented.

    Roman Krzanowski from Verizon said the definition is an open issue; and thinks this draft represents something that is needed. Phil asked which of the things the different tools measured would be most useful (link capacity, available capacity, bottleneck/tight link...); Roman said it depends on the part of the network you were interested in, but he would like to have reliable measurements of capacity. He would especially like agreed upon metrics in this area. Henk said that in discussion with ISPS he finds end-to-end parameters are the most important; ISPs know link parameters, but they don't know end-to-end parameters. Roman has attended the IDQ workshop and available BW was of interest there. Summary will be taken to the list.

    Emile Stephan noted that we have discussed active techniques, but passive techniques are also needed. He said it was important to note what could be measured and with what accuracy. If one operator is sending to another you want to know how the measurement was performed, using what methodology and which accuracy. Roman said he concurred; we have methodology and technology but we do not have agreement.

    7. Discussion on an open network performance testing protocol.
    --Roman Krzanowski Verizon Technologies

    Roman works on architecture and SLA/monitoring, and he is faced with multiple classes/technologies. Despite all the definitions and dedicated capabilities, there are no standards for conducting measurements. The cost for maintenance of measurement devices far exceeds the cost of the device. Scaling is huge at the customer edge - 1000's of points.

    What Roman proposes (desires) is a protocol that is implemented in the deployed hardware for latency, packet loss and jitter. This protocol must be open, so it can be implemented by multiple manufacturers. He also sees the need the measurement designs to be standardized. Outline for protocol requirements (see slides): accuracy <1ms,support the IPPM RFCs, etc.

    Al Morton noted the list of metrics focused on round-trip, which is practical. Roman said yes, because it is easy. You can get accurate time everywhere, but it is not free. Jeff Dunn made a number of observations related to timing and what was possible, including that a system must sample appropriately for the accuracy desired, and these conditions should go into the architecture. Roman said he was interested in the practicality so that we can get something fundamental and basic in everywhere. Jeff also noted you need special treatment of measurement traffic to get accuracy, but that impacts other traffic. Matt Mathis thought we were looking for the holy grail of measurement: you want to measure everything, you want to pay nothing. He said Roman should look at the IPMP draft that has been discussed here previously, and the report from the IRTF measurement group on that protocol. ICMP does work well because it does not get preferred access to resources. IPMP has diverged into two camps, and input from ISPs might help. The IRTF review proposed making this simpler. Just find a way to record a timestamp, and a cookie that the ISP can understand. Can infer everything you are looking for except loss rate from this data. The person from MCI felt that it was very expensive to get core line cards to do the time stamps. It was also important to be sure you define where the edges of what you measure are. Roman will follow up. Keynam from Brix said we shouldn't argue, but be open to all solutions in measurement, both hardware and software. He wanted a clarification if we are looking at a protocol for testing, or a protocol for reporting? Roman said that if you are doing testing, must agree on some way of collecting data, but that is secondary to agreeing on testing methodology and protocol.

    Al Morton said that how the providers make measurements is another point where we need to agree, and we need comparable measurements to do some inter-domain work. We had an AS draft in IPPM that attempted to narrow the field. That might be a good first step toward what you want.

    Roman was going to follow up after reading the IPMP report posted to the IMRG and IPPM list, as well as the OWAMP drafts.

    With that the meeting used the allotted time, and the meeting was closed.

    Slides

    Agenda
    One-way Active Measurement Protocol
    Packet Reordering Metric for IPPM
    Bandwidth Draft
    Open Performance Testing Protocol