Current Meeting Report
Slides


2.7.3 IP Performance Metrics (ippm)

NOTE: This charter is a snapshot of the 53rd IETF Meeting in Minneapolis, MN USA. It may now be out-of-date. Last Modified: 26-Feb-02
Chair(s):
Merike Kaeo <kaeo@merike.com>
Matthew Zekauskas <matt@advanced.org>
Transport Area Director(s):
Scott Bradner <sob@harvard.edu>
Allison Mankin <mankin@isi.edu>
Transport Area Advisor:
Allison Mankin <mankin@isi.edu>
Technical Advisor(s):
Andy Bierman <abierman@cisco.com>
Mailing Lists:
General Discussion:ippm@advanced.org
To Subscribe: ippm-request@advanced.org
Archive: http://www.advanced.org/IPPM/archive/
Description of Working Group:
Note: Andy Bierman serves as MIB advisor.

The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgment (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.

Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.

The IPPM WG will produce documents that define specific metrics and procedures for accurately measuring and documenting these metrics. The metrics are:

- connectivity

- one-way delay and loss

- round-trip delay and loss

- delay variation

- loss patterns

- packet reordering

- bulk transport capacity

- link bandwidth capacity

This is the cumulative set, including the metricsalready completed and published.

The working group will closely review and then be guided by an IESG document on how metrics advance along the standards track within the IETF. This document will also be relevant to the work of the benchmarking working group (BMWG). The first draft of this document was discussed at IETF 51. Additionally, the WG will produce Proposed Standard AS documents, comparable to applicability statements in RFC 2026, that will focus on procedures for measuring the individual metrics and how these metrics characterize features that are important to different service classes, such as bulk transport, periodic streams, or multimedia streams. It is specifically out of scope for this working group to actually characterize traffic, for example to characterize a voice-over-IP stream. Each AS document will discuss the performance characteristics that are pertinent to a specified service class; clearly identify the set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of testing results. Specific topics of these AS documents must be approved by the Area Directors as charter additions.

The WG will produce a protocol to enable communication among test equipment that implements the one-way metrics. The intent is to create a protocol that provides a base level of functionality that will allow different manufacturer's equipment that implements the metrics according to a standard to interoperate. A protocol requirements document will guide the protocol design.

The WG will also produce a MIB to retrieve the results of IPPM metrics, such as one-way delay and loss, to facilitate the communication of metrics to existing network management systems. Thus, the group will create a MIB that contains predominantly read only variables. If, after the protocol requirements document is finished, the group decides that it is appropriate to add variables that control the underlying measurements that the metrics report, such a control structure may be added as a separate document, subject to review by the IESG.

The intent of the WG is to cooperate with other appropriate standards bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote consistent approaches and metrics. Within the IETF process, IPPM metrics definitions will be subject to as rigorous a scrutiny for usefulness, clarity, and accuracy as other protocol standards. The IPPM WG will interact with other areas of IETF activity whose scope intersect with the requirement of these specific metrics. These include working groups such as BMWG, RMONMIB, and TEWG.

Goals and Milestones:
Done   Submit drafts of standard metrics for connectivity and treno-bulk-throughput.
Done   Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC.
Done   Submit documents on delay and loss to IESG for publication as Informational RFCs.
Done   Submit a document on connectivity to IESG for publication as an Informational RFC.
Done   Submit a document on bulk-throughput to IESG for publication as an Informational RFC.
Feb 02   Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC.
Feb 02   Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC.
Feb 02   Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC.
Feb 02   Create initial draft on the definitions of link bandwidth capacity.
Feb 02   First draft for AS on one-way delay and loss.
Mar 02   Create initial draft on a sensitivity analysis of one-way delay and loss metric parameters (companion to the AS).
Mar 02   Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC.
Mar 02   Create initial draft on a MIB for reporting IPPM metrics.
Apr 02   Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document.
Apr 02   Create initial draft on comparing ITU and IETF IP performance metrics.
Apr 02   Create initial draft on a packet reordering metric.
Jun 02   Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC.
Jul 02   Submit draft on bulk transfer capacity metric based on the bulk transfer framework (CAP) to the IESG for Proposed Standard.
Oct 02   Submit recommendation to the IESG on whether to advance, recycle, or deprecate RFCs 2678, 2679, 2680, and 2681 (connectivity, loss, & delay).
Oct 02   Submit draft on a packet reordering metric to the IESG for Proposed Standard.
Dec 02   Submit AS for one-way delay and loss to the IESG for PS.
Dec 02   Submit sensitivity analysis of one-way delay and loss, for consideration as an Informational RFC.
Feb 03   Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard.
Mar 03   Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS.
Mar 03   Discuss rechartering or ending working group.
Internet-Drafts:
Request For Comments:
RFCStatusTitle
RFC2330 Framework for IP Performance Metrics
RFC2678E IPPM Metrics for Measuring Connectivity
RFC2679PSA One-way Delay Metric for IPPM
RFC2680PSA One-way Packet Loss Metric for IPPM
RFC2681PSA Round-trip Delay Metric for IPPM
RFC3148 A Framework for Defining Empirical Bulk Transfer Capacity Metrics

Current Meeting Report

IP Performance Metrics WG (ippm)
Monday, March 18 at 13:00 - 15:00
====================================

The meeting was chaired by the working group chairs, Matt Zekauskas and Merike Kaeo. Al Morton and Tony McGregor took notes, which were edited into these minutes by the chairs.

AGENDA:
1. Agenda bashing
2. Charter Update
3. IPPM MIB discussion
4. Discussion on packet reordering
5. Statistics of One-way Delay
6. IPMP
7. Any remaining outstanding issues

IETF home page: http://www.ietf.org/html.charters/ippm-charter.html
IPPM home page: http://www.advanced.org/IPPM/

3. IPPM MIB discussion
-- Emile Stephan

http://www.ietf.org/internet-drafts/draft-stephan-ippm-mib-02.txt

Emile Stephan presented his work on a "IPPM MIB". The presentation was based upon his current personal submission, which will be split into three parts, two of which will become working group documents. The first will be a metrics definition document that can be shared with RMONMIB MIB objects that wish to use IPPM data. The second will be a "reporting MIB" to present measurement results. This split is as we discussed in Salt Lake City. Andy Bierman will be sending detailed comments to the mailing list to help align the current ideas with the SNMP framework.

A question was asked about whether we needed to finish the OWAMP requirements document first. Merike replied that since we will only be proceeding with the reporting part of the MIB, not control, we did not need to wait for or coordinate with protocol requirements.

Matt asked that given the example of using the MIB on measurement probes, was this a requirement? No, you will be able to use the MIB on a "proxy" that collects results. However, Emile was concerned that you might need to limit the size of history you need to keep, if there was no proxy.

Henk Uijterwaal asked if the MIB allows you to ask for intermediate results of a long-running tests (such as ones that the RIPE Test Traffic machines run).

Henk made the observation that the whole document would be more readable if move section 7 (architecture) to the front rather than discussing MIB itself. It would better show the interrelationships.

There was a question about time filter formats -- it will be taken to the mailing list.

Emile asked what the group thought was more important, to pull the results or to have a filter and push interesting results. Henk noted that the requirement depended on the application; if you are investigating a problem (or a researcher) you might want raw data, where operators would prefer summaries. We should probably cover both.

4. Discussion on packet reordering (20 min)

http://www.ietf.org/internet-drafts/draft-morton-ippm-nonrev-reordering-00.txt
http://www.ietf.org/internet-drafts/draft-critchley-mlas-reordering-00.txt
http://www.ietf.org/internet-drafts/draft-shalunov-reordering-definition-00.txt

Next there were two presentations on packet reordering metrics that covered three personal submissions that occurred since the last IETF meeting. The first was principally on N-reordering, a metric to show how many packets are out of order by a certain amount. The second was on "non-reversing sequences", which captured similar behavior to N-reordering, and in addition added metrics to capture more information about how far packets were out of sequence in dimensions of packets, bytes, and time. The third submission related to "Minimum Longest Ascending Sequences" (MLAS), a technique to give a single number to capture a notion of out of sequence.

N-Reordering
-- Stanislav Shalunov

Mark Allman commented that the metric might be useful to TCP on the fly to adjust TCP's dupack threshold.

Stanislav commented that he thought one problem with MLAS was memory use for long sequences, since it requires O(N) memory.

Another person commented that the interval between probe packets is very important since reordering is dependent on load, and it should be discussed in any document.

Non-reversing Sequences
-- Al Morton

Al felt that just having a "percent reordered" packets was not enough, and the drafts are sufficiently similar that the authors should be able to work together to merge them.

This draft attempts to split the problem into two parts - first, say if reordering has occurred and second, quantify the degree of reordering.

Jeff Richards asked what if a packet is lost or dropped, so that the sequence number is essentially more than one greater than the last? Al replied that was fine, and not considered to be reordering (we have the loss metric for that); any reordering metric should handle lost packets. Matt Z. noted that most router testers today just report out of sequence, so if a packet has a sequence number greater than one more than the last packet, it is counted as out of sequence; if the next packet fills a hole (so the sequence numbers run backward) it too is reported out of sequence. For example, the sequence 1,3,2,4,5 is reported as having three out of sequence packets (3, 2, and 4). This allows them to have one metric to indicate both reordering and loss. Again, Al felt that we already had a loss metric, and we should concentrate on reordering.

There was a discussion as to whether you can have a singleton reordering metric. Stanislav felt that you can only really operate on sequences, Al felt that you do need more than one packet, but that you can define a singleton. They agreed to take the discussion offline.

Someone asked why we care about reordering. Al felt that it happens enough that we should measure it. Later, Matt Mathis pointed out that high-end routers exist that stripe within the router in addition to routers that stripe over parallel connections. A bound for how out-of-order packets can get is a useful result. Linux even has code to do adaptive adjustment of the dupack threshold.

Someone formerly from BBN noted that it is important to map to applications, and that this might require a parameterized metric; a single number was not seen to be useful. For some applications, the number of TCP SACK blocks needed might be the relevant number, but it's useless for others.

Someone from Worldcom liked the time aspects (spacing) of this proposal; we need to illustrate where the time aspects are important.

There was general agreement to merge the three drafts, with the authors working together.

5. Statistics of One-way Delay
-- Dale Pullin

http://www.ietf.org/internet-drafts/draft-corlett-statistics-of-packet-delays-00.txt
http://www.ietf.org/internet-drafts/draft-corlett-statistics-of-packet-delays-00.pdf

Dale Pullin presented on statistics of One-Way Packet Delay. He attempted to find distributions that matched his data. Matt Zekauskas asked whether there were lost packets or whether they were ignored? Dale replied that the national sample data had virtually no packets lost, and the international sample lost 1%. Lost packets were excluded from the analysis.

A discussion ensued regarding the use of gamma distributions. A member felt that gamma functions are good for approximating any function; was there a reason it was specifically well suited for this purpose? Dale's response was that an exponential distribution is a special case of gamma distribution and that the sum of independent exponential will be a gamma variable. However, he stated that maybe the gamma distribution is not completely appropriate for data set over a long time.

Someone commented that perhaps the distributions should be made over like periods during the day; look for stationary periods. Dale disagreed about the cyclic nature over the time periods that were examined. (pdf's were over 300 second periods.)

When looking at the power spectrum over the whole dataset, see peaks at diurnal periods, but the rest is noisy and really shows no structure. Within the frequency range of the chart, there is no indication that there are inherent periods. Perhaps a gaussian filter should be applied to see the true spectral behavior.

6. IPMP
-- Tony McGregor

http://www.ietf.org/internet-drafts/draft-mcgregor-ipmp-00.txt

The final presenter was Tony McGregor who gave a brief history of IPMP and mentioned that one vendor is ready to implement it (not one of 2 biggest ones but it's still a starting point). The outstanding issues are the size of field to which these records are to be inserted, 60 records for now, and what to do with regard to IPv6. However, IPv4 seems to be right place to begin. Tony mentioned the goal was looking for standardization and he felt that the IPPM working group seemed a good place to start.

There was a question as to what the protocol did. Tony explained that there were two parts to the protocol. Routers add timestamps to measurement packets. The protocol allows the router to use any timestamp that it has available - you can find out about the time sync outside of the measurement path. Also, you request from the router what the mechanism is by which you process IPMP packet compared to how you process other packets. It is similar to pathtrace using IP-options. IP options by themselves doesn't work very well since they are very limited in size

Another question concerned Denial of Service attacks and whether this has been thought about. Tony's response was that it created no greater issue than any other IP packet -- it would be processed within the main forwarding path so it doesn't introduce any new opportunities. A comment was made that there would still be a problem when you're querying the CPU but Tony replied that the information request can go low-priority.

Stanislav made a comment that a normal router may look at 6 bits of header and then map all possible values - with IPMP the 6 bits turn into 16 bytes with complex interdependencies. Is that the case? Tony responded that because IPMP is packaged inside IP it would still have the 6 bits. If the router already using different queueing then it would use the protocol field but router only needs to check those things if it already was doing it as part of look-up process.

There was a question of whether IPMP would work with MPLS. There was no clear answer to that yet -- At present MPLS forwarding would work, but would be invisible (like other tunnels). This is not a problem because IPMP doesn't require every router to insert a path record.

There was another comment that there is a policy issue about whether providers would want to be measured. Tony replied that this is true, but some realize that if they will be measured, they want to be measured well. This protocol does not expose any information other than IP addresses, which are usually exposed by traceroutes anyway. There are extensions that allow you to return additional information if you so desire.

The meeting ran out of time, with Matt requesting that if anyone was interested in a protocol like IPMP to show support on the mailing list; both pro and con comments are welcome.

Slides

Agenda
Drafts, Reporting capabilities, and Coupling IPPM-REPORTING-MIB and OWAMP
Defining Packet Reordering Metric
Reordering Metric for IPPM using Non-Reversing Sequence
Statistics of One-Way Internet Packet Delays
Statistics of Internet Packet Delays
IP Measurement Protocol