Current Meeting Report
Slides
2.8.4 IP Performance Metrics (ippm)
NOTE: This charter is a snapshot of the 54th IETF Meeting in Yokohama, Japan. It may now be out-of-date.
Last Modifield: 05/06/2002
Chair(s):
Merike Kaeo <kaeo@merike.com>
Matthew Zekauskas <matt@advanced.org>
Transport Area Director(s):
Scott Bradner <sob@harvard.edu>
A. Mankin <mankin@isi.edu>
Transport Area Advisor:
A. Mankin <mankin@isi.edu>
Technical Advisor(s):
Andy Bierman <abierman@cisco.com>
Mailing Lists:
General Discussion: ippm@advanced.org
To Subscribe: ippm-request@advanced.org
Archive: http://www.advanced.org/IPPM/archive/
Description of Working Group:
Note: Andy Bierman serves as MIB advisor.
The IPPM WG will develop a set of standard metrics that can be applied
to the quality, performance, and reliability of Internet data delivery
services. These metrics will be designed such that they can be
performed by network operators, end users, or independent testing
groups. It is important that the metrics not represent a value
judgment (i.e. define "good" and "bad"), but rather provide unbiased
quantitative measures of performance.
Functions peripheral to Internet data delivery services, such as
NOC/NIC services, are beyond the scope of this working group.
The IPPM WG will produce documents that define specific metrics and
procedures for accurately measuring and documenting these metrics.
The metrics are:
- connectivity
- one-way delay and loss
- round-trip delay and loss
- delay variation
- loss patterns
- packet reordering
- bulk transport capacity
- link bandwidth capacity
This is the cumulative set, including the metricsalready completed
and published.
The working group will closely review and then be guided by an
IESG document on how metrics advance along the standards track within
the IETF. This document will also be relevant to the work of the
benchmarking working group (BMWG). The first draft of this document
was discussed at IETF 51. Additionally, the WG will produce Proposed
Standard AS documents, comparable to applicability statements in RFC
2026, that will focus on procedures for measuring the individual
metrics and how these metrics characterize features that are important
to different service classes, such as bulk transport, periodic streams,
or multimedia streams. It is specifically out of scope for this working
group to actually characterize traffic, for example to characterize a
voice-over-IP stream. Each AS document will discuss the performance
characteristics that are pertinent to a specified service class;
clearly identify the set of metrics that aid in the description of
those characteristics; specify the methodologies required to collect
said metrics; and lastly, present the requirements for the common,
unambiguous reporting of testing results. Specific topics of these
AS documents must be approved by the Area Directors as charter
additions.
The WG will produce a protocol to enable communication among test
equipment that implements the one-way metrics. The intent is to create
a protocol that provides a base level of functionality that will allow
different manufacturer's equipment that implements the metrics
according to a standard to interoperate. A protocol requirements
document will guide the protocol design.
The WG will also produce a MIB to retrieve the results of IPPM
metrics, such as one-way delay and loss, to facilitate the
communication of metrics to existing network management systems. Thus,
the group will create a MIB that contains predominantly read only
variables. If, after the protocol requirements document is finished,
the group decides that it is appropriate to add variables that control
the underlying measurements that the metrics report, such a control
structure may be added as a separate document, subject to review by
the IESG.
The intent of the WG is to cooperate with other appropriate standards
bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote
consistent approaches and metrics. Within the IETF process, IPPM
metrics definitions will be subject to as rigorous a scrutiny for
usefulness, clarity, and accuracy as other protocol standards. The
IPPM WG will interact with other areas of IETF activity whose scope
intersect with the requirement of these specific metrics. These
include working groups such as BMWG, RMONMIB, and TEWG.
Goals and Milestones:
Done | | Submit drafts of standard metrics for connectivity and
treno-bulk-throughput. |
Done | | Submit a framework document describing terms and notions
used in the IPPM effort, and the creation of metrics by the
working group to IESG for publication as an Informational
RFC. |
Done | | Submit documents on delay and loss to IESG for publication
as Informational RFCs. |
Done | | Submit a document on connectivity to IESG for publication
as an Informational RFC. |
Done | | Submit a document on bulk-throughput to IESG for
publication as an Informational RFC. |
FEB 02 | | Submit draft on loss pattern sample metrics to the IESG for
publication as an Informational RFC. |
FEB 02 | | Submit draft on metrics for periodic streams to the IESG
for publication as a Proposed Standard RFC. |
FEB 02 | | Submit draft on IP delay variation to the IESG for
publication as a Proposed Standard RFC. |
FEB 02 | | Create initial draft on the definitions of link bandwidth
capacity. |
FEB 02 | | First draft for AS on one-way delay and loss. |
MAR 02 | | Create initial draft on a sensitivity analysis of one-way
delay and loss metric parameters (companion to the AS). |
MAR 02 | | Submit draft on One-Way Active Measurement Protocol
Requirements to the IESG for consideration as an
Informational RFC. |
MAR 02 | | Create initial draft on a MIB for reporting IPPM metrics. |
APR 02 | | Create draft on a One-Way Active Measurement Protocol that
satisfies the requirements document. |
APR 02 | | Create initial draft on comparing ITU and IETF IP
performance metrics. |
APR 02 | | Create initial draft on a packet reordering metric. |
JUN 02 | | Submit link bandwidth capacity definitions draft to the
IESG, for consideration as an Informational RFC. |
JUL 02 | | Submit draft on bulk transfer capacity metric based on the
bulk transfer framework (CAP) to the IESG for Proposed
Standard. |
OCT 02 | | Submit recommendation to the IESG on whether to advance,
recycle, or deprecate RFCs 2678, 2679, 2680, and 2681
(connectivity, loss, & delay). |
OCT 02 | | Submit draft on a packet reordering metric to the IESG for
Proposed Standard. |
DEC 02 | | Submit AS for one-way delay and loss to the IESG for PS. |
DEC 02 | | Submit sensitivity analysis of one-way delay and loss, for
consideration as an Informational RFC. |
FEB 03 | | Submit draft on a MIB for reporting IPPM metrics to the
IESG for Proposed Standard. |
MAR 03 | | Submit draft on the One-Way Active Measurement Protocol to
the IESG for consideration as a PS. |
MAR 03 | | Discuss rechartering or ending working group. |
Internet-Drafts:
- draft-ietf-ippm-ipdv-09.txt
- draft-ietf-ippm-loss-pattern-07.txt
- draft-ietf-ippm-npmps-07.txt
- draft-ietf-ippm-owdp-04.txt
- draft-ietf-ippm-owdp-reqs-02.txt
- draft-ietf-ippm-metrics-registry-01.txt
- draft-ietf-ippm-reordering-00.txt
- draft-ietf-ippm-reporting-mib-00.txt
- draft-ietf-ippm-owmetric-as-00.txt
Request For Comments:
RFC | Status | Title |
RFC2330 | I | Framework for IP Performance Metrics |
RFC2678 | E | IPPM Metrics for Measuring Connectivity |
RFC2681 | PS | A Round-trip Delay Metric for IPPM |
RFC2680 | PS | A One-way Packet Loss Metric for IPPM |
RFC2679 | PS | A One-way Delay Metric for IPPM |
RFC3148 | I | A Framework for Defining Empirical Bulk Transfer Capacity Metrics |
Current Meeting Report
Minutes of the IP Performance Metrics WG (ippm)
Monday, July 15 at 13:00 - 15:00
====================================
CHAIRS: Merike Kaeo <kaeo@merike.com>, Matt Zekauskas <matt@advanced.org>
Al Morton took notes, which were edited into these minutes by the chairs.
AGENDA:
1. Agenda Bashing
2. Working Group Milestone Status
3. IPPM Reporting MIB and Metrics Registry Discussion
4. Packet Reordering Discussion
5. One-way Delay Protocol Requirements Discussion
6. One-way Metrics Applicability Statement
IETF home page: http://www.ietf.org/html.charters/ippm-charter.html
IPPM home page: http://www.advanced.org/IPPM/
3. IPPM Reporting MIB and Metrics Registry Discussion
-- Emile Stephan
http://www.ietf.org/internet-drafts/draft-ietf-ippm-reporting-mib-00.txt
http://www.ietf.org/internet-drafts/draft-ietf-ippm-metrics-registry-00.txt
Emile Stephan presented on the metrics registry and reporting MIB. First, the
metrics registry document was discussed. The document defines a registry format,
associating OIDs with IPPM metrics. RMON would like the registry so there are
unique, common, ways of referencing the IPPM metrics. One issue yet to be
resolved is how to define new metrics. This draft has a draft tree, where draft
metrics can be placed to avoid ad-hoc assignment. The names would have to be
changed when the metrics move into the "rfc" (standardized) branch. Emile is
also placing metrics from IPPM drafts in the IESG last-call process into the
standardized portion of the registry. Another issue raised is the metric OID
length. He is in discussion with SNMP experts on this issue - one home node for
the IPPM Metrics Registry could allow the metric object identifier to fit in
exactly 8 bytes. There were no comments from the group on these issues.
Next, Emile reported on the status of, and outstanding issues with, the
reporting MIB. The first major issue is with timestamp resolution. His current
proposal uses timestamps with 250 picosecond resolution. Is that enough for the
next few years?
Henk Uijterwaal made the comment that the proposed timestamp looks like it has
the year 2100 problem - only two digits are displayed. He advocated using
timestamps from NTP; then have multiple resolutions and all the tools exist for
conversions. Jon Bennet supported this idea. Emile made a comment that you need
something that could be used as an index, and you must have GMT not local time.
You also need something human readable. Henk noted that NTP timestamps are GMT:
tools exist for human-readable forms (also to convert from GMT to some local
reference) and that as a 32 bit number it should be a suitable index. Emile will
look into the issue, and there will be further discussion on the mailing list.
Next, Emile noted that he added a field for synchronization (is clock
synchronized or not?). A question from the floor asked about accuracy - Emile
responded that the accuracy was in the MIB. Measurement management issues were
raised. Is it useful to have history when one measure was more precise than
other? Some seemed to agree that this was useful. Emile proposed SNMP over TCP
as one way to help secure connections, so that you could control the
measurements; he also made the point that it is important to have measurement
packet interoperability. Matt Z. made statement that giving Emile feedback on
control is fine, but it's not what we're doing in the working group. Al Morton
noted that packet format is exactly what OWAMP is addressing.
4. Discussion on Packet Reordering
-- Al Morton
http://www.ietf.org/internet-drafts/draft-ietf-ippm-reordering-00.txt
Al Morton presented status of current draft: it is a combination of the three
drafts presented at the previous IETF. MLAS needs complete sequence and doesn't
work if packets are lost, which is problematic, so it is not considered in this
draft. (MLAS did have good property of identifying how many steps needed to
restore order; it is just not clear how to apply it). The authors decided on two
goals: first, determine whether packet order is maintained (and which packets
are reordered). Second, quantify the extent of change.
Al outlined the changes yet to be made, the principle one being the use of
consistent notation. He also went over an extensive example showing the metrics
in use (one comment was to make sure the examples in the draft show the
application of all the metrics). Stanislav noted that the vector of arrival
order in the example shown on the slides was not represented correctly - the
notation should be fixed. Notation in the example section of the document should
be checked too.
Merike asked how many people read draft; about 15 people raised hands. There
were no other substantive comments from the group. Merike asked people to take
and further comments to mailing list!
5. Discussion on One-Way Active Measurements Protocol Requirements -- Stanislav
Shalunov
http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-reqs-02.txt
see also http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-04.txt
Stanislav Shalunov presented the modifications to the OWAMP requirements
document. In general, there were just textual modifications and small
clarifications. Stanislav noted that the authors thought it was ready for
publication. Matt took a rough consensus poll: approximately 5 had read the
draft, and of those, all thought it should be moved forward. As a check Matt
asked how many were interested; in the room approximately 10 raised their hands
(most of the rest appeared to be lurkers). Al asked if the timestamp resolution
comments from December were taken into account. Stanislav said that the
timestamp resolution itself does not belong in requirement document. After
discussion, Al was satisfied, but will reread the draft. Matt said the chairs
would call for WG last call after the IETF.
Stanislav then presented the changes in latest protocol draft, as well as
pending changes. He asked for comments on pending changes before they actually
go in the next version of the draft.
A person from the audience (Sharam) asked a general question: why not use ICMP
time stamps with a reliable return channel? You can then do everything from just
the source. Stanislav pointed out that in abstract there was no problem with
ICMP packets (as long as you have synchronized clocks), but you can't use them
for accurate performance measurement. ICMP is not reliable, not treated like
other packets in all networks, it may be identifiable, and it needs processing
time at the responder (which may affect accuracy). With ICMP you can also not
guarantee that the forward path is the same as the reverse path. The questioner
mentioned that perhaps there could be some kind of traffic-engineered reverse
path to "ensure" reliability.
6. One-Way Metric Applicability Statement -- Henk Uijterwaal
http://www.ietf.org/internet-drafts/draft-ietf-ippm-owmetric-as-00.txt
Finally, Henk Uijterwaal led a discussion of open issues with the one-way metric
applicability statement draft. He started by noting that the draft should have a
negative version number - it attempts to lay out the issues, but doesn't provide
lots of answers. The intent is to use it as a basis for discussion; input is
especially requested from other operators.
The first issue was sending rate. Stanislav noted that the rate really depends
on what you want to measure; looking for delay, one value is appropriate, for
reordering you would want the rate to match a target application, and for
capacity you want a much higher rate. He didn't see any single answer. (There
was a "3% of link capacity discussion, which will be reported in a subsequent
paragraph.) John Bennett noted that the rate needed might vary by time of day as
well. Henk agreeded, but noted that bandwidth measurements certainly could swamp
a link, although it might be appropriate if done infrequently.
The next issue that generated discussion was packet size. Al Morton and
Stanislav Shalunov were among the contributors. Again, the size depends on the
purpose. IPDV requires one packet size, but that's only for a particular
variation; there's no reason you can't calculate IPDV on different sized packet
streams.
Another issue was test duration. For delays, can do things continuously; for
bandwidth you need to be careful and the test should be short-term. Al Morton
noted that how frequently you report intermediate results determines how quickly
you can act on it. He has a system that does this every 15 minutes, and that
seems to be a good compromise. There might be some data loss in-between cycles,
but it is a way to get fairly good coverage with a compromise.
Another issue is data volume. 3% of link capacity was a figure from questioning
providers. Note that it is a MAXIMUM over a LONG TERM. We are looking for
providers that might violently disagree. You might also need to specify a
distribution instead of just a blanket volume.
The final issue that generated discussion for this document was security. Henk
noted that there are potential DDoS problems: any single stream might be
acceptable, but the total measurements might swamp a single receiver. (For
situations when there are many senders to one receiver.) Stanislav mentioned
that any kind of "amplifier" is not acceptable. For OWAMP, we're looking at
authentication, and if a measurement point doesn't know you, then it will only
send traffic back to you and not accept traffic from you.
Finally, a group member asked about packet loss and errors. In the wireless
area, very interested in loss due to errors instead of loss due to congestion.
Was there a differentiation with IPPM? If you are looking at the network, you
often don't have any observability into why a packet was lost. This question
might be better addressed in tewg; Stanislav noted that the solution is sub-ip,
so perhaps the measurement should be in sub-ip.
Slides
Agenda
IPPM Reporting MIB & Registry
Packet Reordering Metric for IPPM
- Al Morton
- Len Ciavattone
- Gomathi Ramachandran
- Stanislav Shalunov
- Jerry Perser
A One-way Active MEasurement Protocol
- Stanislav Shalunov
- Ben Teitelbaum
- Matt Zekauskas
One Way Metric Applicability Statement