2.7.7 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 63rd IETF Meeting in Paris, France. It may now be out-of-date.

Last Modified: 2005-06-27

Chair(s):

Henk Uijterwaal <henk@ripe.net>
Matthew Zekauskas <matt@internet2.edu>

Transport Area Director(s):

Allison Mankin <mankin@psg.com>
Jon Peterson <jon.peterson@neustar.biz>

Transport Area Advisor:

Allison Mankin <mankin@psg.com>

Technical Advisor(s):

Andy Bierman <ietf@andybierman.com>

Mailing Lists:

General Discussion: ippm@ietf.org
To Subscribe: https://www1.ietf.org/mailman/listinfo/ippm
In Body: subscribe
Archive: http://www.ietf.org/mail-archive/web/ippm/index.html

Description of Working Group:

Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied
to the quality, performance, and reliability of Internet data delivery
services. These metrics will be designed such that they can be
performed by network operators, end users, or independent testing
groups. It is important that the metrics not represent a value
judgment (i.e. define "good" and "bad"), but rather provide unbiased
quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as
NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and
procedures for accurately measuring and documenting these metrics.
The metrics are:

  - connectivity

  - one-way delay and loss

  - round-trip delay and loss

  - delay variation

  - loss patterns

  - packet reordering

  - bulk transport capacity

  - link bandwidth capacity

This is the cumulative set, including the metricsalready completed
and published.

The working group will closely review and then be guided by an
IESG document on how metrics advance along the standards track within
the IETF.  This document will also be relevant to the work of the
benchmarking working group (BMWG).  The first draft of this document
was discussed at IETF 51. Additionally, the WG will produce Proposed
Standard AS documents, comparable to applicability statements in RFC
2026, that will focus on procedures for measuring the individual
metrics and how these metrics characterize features that are important
to different service classes, such as bulk transport, periodic
streams,
or multimedia streams. It is specifically out of scope for this
working
group to actually characterize traffic, for example to characterize a
voice-over-IP stream. Each AS document will discuss the performance
characteristics that are pertinent to a specified service class;
clearly identify the set of metrics that aid in the description of
those characteristics; specify the methodologies required to collect
said metrics; and lastly, present the requirements for the common,
unambiguous reporting of testing results. Specific topics of these
AS documents must be approved by the Area Directors as charter
additions.

The WG will produce a protocol to enable communication among test
equipment that implements the one-way metrics. The intent is to create
a protocol that provides a base level of functionality that will allow
different manufacturer's equipment that implements the metrics
according to a standard to interoperate. A protocol requirements
document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM
metrics, such as one-way delay and loss, to facilitate the
communication of metrics to existing network management systems. Thus,
the group will create a MIB that contains predominantly read only
variables.  If, after the protocol requirements document is finished,
the group decides that it is appropriate to add variables that control
the underlying measurements that the metrics report, such a control
structure may be added as a separate document, subject to review by
the IESG.

The intent of the WG is to cooperate with other appropriate standards
bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote
consistent approaches and metrics. Within the IETF process, IPPM
metrics definitions will be subject to as rigorous a scrutiny for
usefulness, clarity, and accuracy as other protocol standards. The
IPPM WG will interact with other areas of IETF activity whose scope
intersect with the requirement of these specific metrics. These
include working groups such as BMWG, RMONMIB, and TEWG.

Goals and Milestones:

Done  Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done  Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done  Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done  Submit a document on connectivity to IESG for publication as an Informational RFC.
Done  Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
Done  Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
Done  Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
Done  Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
Done  First draft for AS on one-way delay and loss.
Done  Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
Done  Create initial draft on a MIB for reporting IPPM metrics.
Done  Create initial draft on a packet reordering metric.
Done  Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
Done  Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
Oct 04  Submit draft on a packet reordering metric to the IESG for Proposed Standard.
Dec 04  Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
Dec 04  Collect implementation reports for RFCs 2678-2681
Jan 05  Create initial draft on the definitions of link bandwidth capacity.
Feb 05  Discuss rechartering or ending working group.

Internet-Drafts:

  • draft-ietf-ippm-owdp-14.txt
  • draft-ietf-ippm-metrics-registry-08.txt
  • draft-ietf-ippm-reordering-10.txt
  • draft-ietf-ippm-implement-00.txt
  • draft-ietf-ippm-bw-capacity-00.txt

    Request For Comments:

    RFCStatusTitle
    RFC2330 I Framework for IP Performance Metrics
    RFC2678 E IPPM Metrics for Measuring Connectivity
    RFC2679 PS A One-way Delay Metric for IPPM
    RFC2680 PS A One-way Packet Loss Metric for IPPM
    RFC2681 PS A Round-trip Delay Metric for IPPM
    RFC3148 I A Framework for Defining Empirical Bulk Transfer Capacity Metrics
    RFC3357 I One-way Loss Pattern Sample Metrics
    RFC3393 PS IP Packet Delay Variation Metric for IPPM
    RFC3432 PS Network performance measurement for periodic streams
    RFC3763 I A One-way Active Measurement Protocol Requirements

    Current Meeting Report

    IP Performance Metrics WG (ippm)
    Monday, August 1, 2005 - 10:30--12:30
    =====================================
    
    The meeting was chaired by Henk Uijterwaal and Matt Zekauskas.  Dave Plonka
    and Al Morton took notes, which were edited into these minutes by the
    chairs.
    
    
    AGENDA:
    
    1. Administrivia: Agenda, Status of Drafts and Milestones
    2. Implementation reports update
        draft-ietf-ippm-implement-00.txt
    3. Capacity draft
        draft-ietf-ippm-bw-capacity-00.txt
    4. Reordering
        draft-ietf-ippm-reordering-10.txt
    5. Traceroute storage
        draft-niccolini-ippm-storetraceroutes-01.txt
    6. Multi-point metrics
        draft-stephan-ippm-multimetrics-01.txt
    7. Composition of metrics
        draft-morton-ippm-composition-00.txt
    8. Two-way Active Measurement Protocol
        draft-hedayat-two-way-active-measurement-protocol-00.txt
    
    
    
    1. Administrivia: Agenda, Status of Drafts and Milestones
    
    Henk Uijterwaal opened up the meeting by reviewing the agenda, and
    then the status of outstanding drafts (see slides).  There were no
    audience requests to change the agenda (although the multimetrics and
    composition requests were flipped at the request of the speakers
    later, and the eighth item was added just before the meeting).  The
    metrics registry is in AUTH48; The OWAMP draft has been submitted to
    the IESG, there have been some minor changes in response to IESG
    comments (including minor on-the-wire changes); the draft is currently
    being discussed with the security AD; the authors are currently
    waiting for feedback; other drafts are discussed below.
    
    
    
    
    
    2. Henk Uijterwaal: Implementation reports update
        draft-ietf-ippm-implement-00.txt
       
    Henk then segued into the implementation reports draft.  Henk has some
    small updates pending for the draft, then it appears to be done.  Matt
    said he would try to get another system (Internet2's "owamp" code) into
    the draft although there appear to be enough implementations to recommend
    advancement.  The chairs will discuss with the proper disposition of
    this draft with the AD.
    
    
    
    
    3. Matt Zekauskas for Phil Chimento and Joe Ishac: Capacity draft
        draft-ietf-ippm-bw-capacity-00.txt
    
    Matt gave a presentation by Phil Chimento and Joe Ishac on the
    capacity draft.  The purpose of this draft is to provide clear
    and precise capacity definitions.  Two outstanding issues brought
    up on the mailing list along with proposed changes (also on the
    mailing list) were shown.
    
    Matt personally expressed happiness with the direction of the current
    draft.  
    
    Al Morton asked about the concept of "valid IP bits" - what does this mean
    for payload?  It assumes that a simple match of byte count; Al doesn't think
    this is enough [to say its valid].  Matt said that he thought that since
    we are showing capacity and utilization, any bits are probably fine.
    Al suggested comparing the definition with the rest of the framework.
    
    Emile Stephan wondered if we would have some metrics as the result of
    this draft.  Matt stated that he thought we *could*, following the
    precedent of bulk measurement.  Emile thought that what was proposed
    was massively intrusive and the capacity definitions are not compatible
    with the framework.  Phil (via Jabber) stated that the document is
    only there to set definitions.
    
    Farooq Bari asked if the authors intended to do anything above IP
    (such as TCP).  Matt said that he believes no; Phil confirmed via
    jabber that the current document is only intended to be definitions.
    The same audience member thought it might be interesting to see TCP
    capacity addressed because of retransmissions.
    
    
    
    
    4. Al Morton: Update on reordering metric draft
        draft-ietf-ippm-reordering-10.txt
    
    Al Morton gave an update on the reordering draft, and walked through
    a histogram example to show the difference between the proposed
    byte offset with the Colorado State reordering density metric
    (in response to a PDF posted by Nischal Piratla to the list the
    night before the meeting). 
    
    The comparison was a direct comparison between Reordering Byte Offset
    in the current draft, compared with Reordering Buffer Density (RBD).  Al
    felt that Byte Offset is more directly focused on reordering, since
    it gives the buffer size that would accommodate different proportions
    of reordered packets, while RBD gives buffer occupancy over all
    packets in the stream.  
    
    About five people in the audience raised their hands when Al asked
    who had read the draft.
    
    Henk asked how we should proceed with this draft?  All comments have
    been respond to except the PDF posted last night.  Should we declare
    it finished and move to last call?  There was no audience response.
    Henk said he would ask again on the mailing list.  Al said that
    in summary, many people have read the draft, and said they were mostly
    satisfied, and any comments that were made were incorporated.  He feels
    that the comment last night is a comparison between the metrics, not
    saying what we're doing is wrong, but others may be better in some
    respects.  Most are happy with what we have, and a few folks have
    other ideas.  Henk agreed with the summary, and felt that there was
    rough consensus now to move to Last Call.  [Note: on Jabber,
    Mark Allman asked how the byte offset and RBD metrics "square"; Henk
    had closed the topic and said the question would be taken off-line.]
    
    
    
    
    
    
    5. Juergen Quittek - Traceroute Metrics and Data Model (see slides)
        draft-niccolini-ippm-storetraceroutes-01.txt
    
    Juergen Quittek talked about a traceroute data model and exchange
    format that they would like to see standardized (hence taken up
    by the working group).  They surveyed other measurement repositories;
    CAIDA has 7 years of data, but said they suggested that their
    format not be used.  The proposed works is more of an exchange format
    than a metric; the draft is now in it's second revision.  The names
    in the information model align with the other traceroute work in
    the IETF in the DISMAN traceroute MIB.  So far they prefer an XML
    encoding.
    
    Henk asked for comments.  Jerome Durand said that the Global Grid Forum
    Network Measurement Working Group had common ways to represent metrics
    in XML.  Juergen said that he wasn't aware of the work; Jerome said he
    would post a pointer to the list.  (Comment from Matt: the GGF operates
    in a manner similar to the IETF.)
    
    Emile Stephan thought that a common representation for traceroute was
    a good idea, and he wondered why we don't use an IPFIX representation
    rather than ASCII/XML (i.e. use the IPFIX data model).  Juergen stated
    that it is not necessarily obvious to use passive IPFIX representation
    for an active traceroute measurement, but agreed it is an open point.
    
    Emile also asked if there is already a MIB, what is the purpose of
    this document?  Juergen responded that the MIB doesn't suggest how
    to store it locally, nor how to send the results to a location other
    than where the measurement originated.
    
    Juergen also noted that generalizing IPFIX was denounced by the ops ADs
    (in light of current workload).  Juergen asked if it could be made a working
    group item.
    
    Someone on jabber asked how to interpret the symbols on slide 6:
    + means well suited, - means not suited, 0 means "don't care".
    
    Matt Mathis felt that this would be useful to find landmarks for measuring
    performance in multi-provider environments.  He believes this work is critical,
    and should be done somewhere.  He mentioned that he was aware of the GGF work,
    but it might be useful to do the work twice as a sanity check.  He just wanted
    to make an observation, not advocate the work be done in the WG at this
    time.
    
    Dave Plonka felt that transport of traceroute is probably in scope for IPPM,
    but his inclination is to say storage format is not.  He also said that 
    the IPFIX working group will touch on it this afternoon.
    
    Matt and Henk both echoed that they felt it is unclear whether the storage
    (serialization) format is in scope.  Henk agreed that it is important
    that this work is done somewhere.  Emile thought that there should be
    a discussion with PSAMP and IPFIX and see if there was a general solution
    for the export of measurement data.  Juergen said that PSAMP and IPFIX
    have a binary export format;  IPFIX does not define how to store the info,
    i.e. files are not exchangeable (necessarily), which is why they didn't go
    to IPFIX.    Matt thought that there was general agreement the work should
    be done somewhere, but the question is where.
    
    
    
    
    6. Lei Lang: Multi-point metrics
        draft-stephan-ippm-multimetrics-01.txt
    
    Lei Liang gave a presentation provocatively titled "IPPM Metrics for
    IPTV performance and QoS measurement" (see slides).  Simplistically,
    the multimetrics draft looks at broadcast multicast streams, and
    reports on performance, either "one-to-many" (where you report on
    end-to-end results) or "spatial" (where to also report on performance
    to intermediate points) (which could also be used for a unicast
    stream).  They used broadcast IPTV as a motivator.
    
    Keyam Hedayat said that he had not read the draft, but one of the most
    important IPTV metrics is channel change time.  Many people claim
    proprietary solutions for < less than 10ms change times for example;
    this has to do with multicast join performance.  Have you considered
    this is a metric?  Lei said that they had not considered this.
    Emile thought such a metric was very needed; he wondered if one
    could start with a BMWG multicast benchmark.
    
    Farooq Bari asked if the spatial metrics on slide 5 could be defined in
    generic form, not specific to IPTV?  If so, why not do that instead?
    Why would IETF want to standardize service-specific metrics for
    QoS, if general ones are sufficient?   Lei was initially confused
    by the question; Matt restated as: "why are you developing an
    IPTV-specific metric, rather than something general?";  Matt then
    answered his own question, and said that from the presentation, they
    are general metrics and IPTV was just the motivation.   Lei agreed.
    Someone else said that we should be careful about using the term QoS,
    since it has a specific other meaning(s) elsewhere in IETF.
    Emile said that to clarify: this work defines two kinds of metrics,
    spatial and also one-to-group metrics.
    
    Al Morton noted that with respect to channel changing time, the IGMP
    join flows up tree toward source, and this work assumes there is a
    single source packet going down the path, so that may be out of scope.
    
    Matt Mathis thought this seems service provider oriented, and one way
    (one-to-many), not symmetric.  He wondered, especially in the one-to-group
    case is there a straightforward way to generalize it for "subscriber use":
    many-to-one.  For example, measuring the best path (say based on loss
    rate) from a large number of web servers to a user.  Lei thought it
    could be done. Matt Mathis  asked for clarification: do you mean
    group-to-one measurements will be easy to define, but they are not
    in the draft?  When evaluating service providers, that would be useful.
    
    There was concern that 1-many might not be part of the IPPM framework,
    which usually follows a single source packet through the network.
    Matt Mathis noted that the time interval you care about might be 
    hours or a day... if shopping for an ISP, you want to know how well
    connected to the rest of the world it is.  If the time interval was
    a day, you might not care about synchronization.
    
    Al asked if this is of interest to IPPM?  Henk deferred that question
    until after the next presentation, which seems related.
    
    
    
    
    
    7. Al Morton: Composition of Metrics
        draft-morton-ippm-composition-00.txt
    
    Al Morton gave a presentation on composing metrics -- multiple
    independent measurements along a given path (see slides).
    Concatenation in space and aggregation in space and time was
    considered.  (Al noted that he wrote the draft in an evening; many
    ideas came from the five co-authors listed at the front of the draft,
    but he did not have a chance to review the draft with them before the
    -00 deadline.)  The presentation co-author (Maurizio Molina) is active
    in the GEANT2 Joint Research Activity devoted to measurement (JRA1),
    and they have a milestone on metric composition.  JRA1 has
    concentrated more on temporal composition; this draft focuses mostly
    on spatial composition.
    
    Al asked if there interest to take this up as a work item?
    He see this as a separate item from the previous multimetrics presentation.
    
    Emile said that he doesn't agree with the average summation example, but
    won't make a big issue at this time.  He asked for clarification,
    are we only going to define spatial or both spatial and temporal
    compositions?  Al said that this draft just focuses on spatial
    composition; JRA1 is working on temporal composition.
    
    Matt Mathis thought his sort of work is exactly what the group was
    thinking of with the AFRAME concept in the framework.  One obvious
    metric that is missing is reordering... the mathematics of compositions
    of reordering is complex.  It may be too complex to add to this composition
    work.  May need to separate metrics that have certain engineering properties
    and others that have certain mathematical properties.   There is a fundamental
    difference between the composition of symmetrical and non-symmetrical
    metrics.  Reordering (in TCP, for instance) is not symmetrical with
    respect to composition.  This may be useful to think through to formalize
    the metrics you wish to.  Al also mentioned steady-state versus transient
    reordering; composition is likely easier with a steady-state assumption. 
    
    Matt also mentioned a related problem - what data thinning does.  Say
    two streams mix, the aggregate is reordered, and then the two streams
    separate.  What does this mean?
    
    Al said he left reordering out of the draft because he still feels it's
    a research topic.
    
    Henk felt that there is interest in both the previous work (Lei's
    presentation) and this.  We will ask the mailing list, but thinks
    the group should pick up; the chairs will talk to the AD about adding
    milestones.
    
    Steven Van Den Berghe felt it would be nice to have a new framework,
    and fit both drafts into this composition framework.  He thought the
    same issues will be recurring in both drafts.  Al said that right now
    there isn't much framework (for spatial); it could be copied to both
    drafts.  Steven was encouraged to send in a text contribution.
    
    
    
    
    
    8. Kaynam Hedayat: Two-way Active Measurement Protocol (no slides)
        draft-hedayat-two-way-active-measurement-protocol-00.txt
    
    Kaynam gave a brief update on the TWAMP draft.  He said that
    the draft itself hasn't changed, although he just reissued it
    in anticipation of being a working group document.  There
    has been an implementation of the draft; if anyone out there
    has another implementation and would like to test interoperation
    contact Kaynam.
    
    With that, the meeting was closed.
    

    Slides

    Agenda, Milestone and Implementation Report Update
    Capacity Draft
    Packet Reordering Metric for IPPM
    Traceroute Metrics and Data Model
    Multiparty Metrics ("IPPM Metrics for IPTV Performance and QoS Measurement")
    Composition of Metrics