Last Modified: 2003-02-19
The IPPM WG will develop a set of standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services. These metrics will be designed such that they can be performed by network operators, end users, or independent testing groups. It is important that the metrics not represent a value judgment (i.e. define "good" and "bad"), but rather provide unbiased quantitative measures of performance.
Functions peripheral to Internet data delivery services, such as NOC/NIC services, are beyond the scope of this working group.
The IPPM WG will produce documents that define specific metrics and procedures for accurately measuring and documenting these metrics. The metrics are:
- connectivity
- one-way delay and loss
- round-trip delay and loss
- delay variation
- loss patterns
- packet reordering
- bulk transport capacity
- link bandwidth capacity
This is the cumulative set, including the metricsalready completed and published.
The working group will closely review and then be guided by an IESG document on how metrics advance along the standards track within the IETF. This document will also be relevant to the work of the benchmarking working group (BMWG). The first draft of this document was discussed at IETF 51. Additionally, the WG will produce Proposed Standard AS documents, comparable to applicability statements in RFC 2026, that will focus on procedures for measuring the individual metrics and how these metrics characterize features that are important to different service classes, such as bulk transport, periodic streams, or multimedia streams. It is specifically out of scope for this working group to actually characterize traffic, for example to characterize a voice-over-IP stream. Each AS document will discuss the performance characteristics that are pertinent to a specified service class; clearly identify the set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of testing results. Specific topics of these AS documents must be approved by the Area Directors as charter additions.
The WG will produce a protocol to enable communication among test equipment that implements the one-way metrics. The intent is to create a protocol that provides a base level of functionality that will allow different manufacturer's equipment that implements the metrics according to a standard to interoperate. A protocol requirements document will guide the protocol design.
The WG will also produce a MIB to retrieve the results of IPPM metrics, such as one-way delay and loss, to facilitate the communication of metrics to existing network management systems. Thus, the group will create a MIB that contains predominantly read only variables. If, after the protocol requirements document is finished, the group decides that it is appropriate to add variables that control the underlying measurements that the metrics report, such a control structure may be added as a separate document, subject to review by the IESG.
The intent of the WG is to cooperate with other appropriate standards bodies and forums (such as T1A1.3, ITU-T SG 12 and SG 13) to promote consistent approaches and metrics. Within the IETF process, IPPM metrics definitions will be subject to as rigorous a scrutiny for usefulness, clarity, and accuracy as other protocol standards. The IPPM WG will interact with other areas of IETF activity whose scope intersect with the requirement of these specific metrics. These include working groups such as BMWG, RMONMIB, and TEWG.
Done | Submit drafts of standard metrics for connectivity and treno-bulk-throughput. | |
Done | Submit a framework document describing terms and notions used in the IPPM effort, and the creation of metrics by the working group to IESG for publication as an Informational RFC. | |
Done | Submit documents on delay and loss to IESG for publication as Informational RFCs. | |
Done | Submit a document on connectivity to IESG for publication as an Informational RFC. | |
Done | Submit a document on bulk-throughput to IESG for publication as an Informational RFC. | |
FEB 02 | Submit draft on loss pattern sample metrics to the IESG for publication as an Informational RFC. | |
FEB 02 | Submit draft on metrics for periodic streams to the IESG for publication as a Proposed Standard RFC. | |
FEB 02 | Submit draft on IP delay variation to the IESG for publication as a Proposed Standard RFC. | |
FEB 02 | Create initial draft on the definitions of link bandwidth capacity. | |
FEB 02 | First draft for AS on one-way delay and loss. | |
MAR 02 | Create initial draft on a sensitivity analysis of one-way delay and loss metric parameters (companion to the AS). | |
MAR 02 | Submit draft on One-Way Active Measurement Protocol Requirements to the IESG for consideration as an Informational RFC. | |
MAR 02 | Create initial draft on a MIB for reporting IPPM metrics. | |
APR 02 | Create draft on a One-Way Active Measurement Protocol that satisfies the requirements document. | |
APR 02 | Create initial draft on comparing ITU and IETF IP performance metrics. | |
APR 02 | Create initial draft on a packet reordering metric. | |
JUN 02 | Submit link bandwidth capacity definitions draft to the IESG, for consideration as an Informational RFC. | |
JUL 02 | Submit draft on bulk transfer capacity metric based on the bulk transfer framework (CAP) to the IESG for Proposed Standard. | |
OCT 02 | Submit recommendation to the IESG on whether to advance, recycle, or deprecate RFCs 2678, 2679, 2680, and 2681 (connectivity, loss, & delay). | |
OCT 02 | Submit draft on a packet reordering metric to the IESG for Proposed Standard. | |
DEC 02 | Submit AS for one-way delay and loss to the IESG for PS. | |
DEC 02 | Submit sensitivity analysis of one-way delay and loss, for consideration as an Informational RFC. | |
FEB 03 | Submit draft on a MIB for reporting IPPM metrics to the IESG for Proposed Standard. | |
MAR 03 | Submit draft on the One-Way Active Measurement Protocol to the IESG for consideration as a PS. | |
MAR 03 | Discuss rechartering or ending working group. |
RFC | Status | Title |
---|---|---|
RFC2330 | I | Framework for IP Performance Metrics |
RFC2678 | E | IPPM Metrics for Measuring Connectivity |
RFC2679 | PS | A One-way Delay Metric for IPPM |
RFC2680 | PS | A One-way Packet Loss Metric for IPPM |
RFC2681 | PS | A Round-trip Delay Metric for IPPM |
RFC3148 | I | A Framework for Defining Empirical Bulk Transfer Capacity Metrics |
RFC3357 | I | One-way Loss Pattern Sample Metrics |
RFC3393 | PS | IP Packet Delay Variation Metric for IPPM |
RFC3432 | PS | Network performance measurement for periodic streams |
IP Performance Metrics WG (ippm) Tuesday, March 18, 2003 at 15:45 to 16:45 ========================================= The meeting was moderated by the working group chairs, Matt Zekauskas and Merike Kaeo. Dave McDysan took notes, which were edited into these minutes by the chairs. Thanks also to Simon Leinen, who scribed the meeting into Jabber; some of his notes were used while generating this text. AGENDA: 1. Agenda bashing, Working Group Milestones Status 2. One-way Metric Applicability Statement 3. Discussion on One-Way Active Measurements Protocol 4. Discussion on Packet Reordering 5. IPPM Reporting MIB 6. IPPM Metrics Registry 1. Agenda bashing & Working Group Milestones Status There were no suggestions to modify the agenda. Slides presented by Matt Z noted a new RFC, and overviewed the current work. Matt called for Implementation Reports to move to existing metrics to draft standard - send Email to a chair if you know of an implementation. (Advancement requires an advancing metrics RFC approved by the transport ADs and the IESG first, however; that work is currently in the Transport WG.) Matt noted there were a number of milestones that had no progress. If no interest is expressed, they will be dropped from the charter. We are looking for people to contribute on the following topics (assuming the group still feels they are worthwhile): * parameter sensitivity * ITU (e.g., Y.1540) vs IETF metrics Al Morton indicated that these are similar, recommend dropping this item from the charter * Path bottleneck definitions, * CAP (or other BTC metric) 2. One-way Metric Applicability Statement --Henk Uijterwaal Henk noted that there were no comments on the draft since the Atlanta meeting, and that perhaps it just wasn't time to do this work. However, Merike has been working to see if there was interest in the provider community, and today Henk and Merike met with people from two providers and they were interested in moving the work forward. They were looking for specific questions; Henk and Merike will identify areas where input is needed from people, and they will send questions off to providers. They would welcome help. 3. Discussion on One-Way Active Measurements Protocol -- Stanislav Shalunov First, the requirements document. http://www.ietf.org/internet-drafts/draf t-ietf-ippm-owdp-reqs-06.txt The requirements document passed WG last call, and IESG comments are being addressed. Steve Bellovin was worried about replay attacks; but in some sense that is what we want to measure... we would like to measure duplication, which may or may not be malicious. Similarly, an attacker could add delay and we would like to measure that as well. We do have a requirement that traffic is hard to spot, for example there should be no fixed port number. It would be non-trivial to place OWDP traffic in a separate queue. Allison Mankin came to the mike as AD. She mentioned that authorization text for control protocol design needs to be strengthened - it needs to be more than IP address per current draft. OWAMP needs strong authentication for users since it is a powerful protocol. Anonymous identity wording was also problematic. Stanislav clarified that cryptographic authentication implementation is required; however, a user could request a less secure authentication (e.g., based on IP addresss) Allison, Stanislav, Merike to agree on wording -- could be along lines of clarifying mandatory implementation but optional invocation in the security considerations section. Next, we moved to the current OWAMP protocol specification. http://www.ietf.org/internet-drafts/draf t-ietf-ippm-owdp-06.txt There are changes in the current draft based upon implementation experience, they are shown in the slides. Three new items: First, advertise server uptime at the beginning of each connection (helps with long-term monitoring) -- this disambiguates between whether the other end crashed or the network failed. Second, stop-sessions now supplies number of packets sent in each appropriate session to help decide if the last packet was lost. Third, the algorithm to compute exponentially distributed pseudo-random deviates was documented, and sample code supplied to ensure that all implementations generate the same sequence. Vectors will be added to allow code testing. 4. Discussion on Packet Reordering -- Al Morton http://www.ietf.org/internet-drafts/draf t-ietf-ippm-reordering-02.txt As background, group asked to review Stanislav's n-reordering draft. http://www.ietf.org/internet-drafts/draf t-shalunov-reordering-definition-02.txt There was an independent draft from Colorado State also submitted, but none of the authors were at meeting. A minority of people in the room had read the draft. Al gave a 'tutorial' on reordering. He noted that the independent draft -- a "density" metric -- took into account lost packets, and was therefore not acceptable. Out of order packets can be viewed as "early" or "late". "Late" packets trigger reordering computation at the receiver. We had trouble merging all metrics to section 4.2, in particular we could not include Stanislav's n-reordering without change. However, there was significant editing to unify notation. A "gap" metric was added (section 4.4) based on a input from Jon Bennett, as the distance between the beginning of reordering events. Matt Mathis asked how this handle nested reordering events caused by multiple reordering events. e.g., the arrival is 1,2,4,5,6,7,9,10,11,8,3,12. Jon stated that this metric might be trying to keep track of too much state and structure. Authors to think about this and discuss. Al proposed to fix the problem (with merging the metrics) in the draft by reorganization. Keep section 3 (determining whether or not packet order is maintained), then to quantify the extent of the changes split into two sections: Sec 4 - Metrics leaning to network characterization (frequency, distance, and a metric for multiple events (e.g., reordering gap). Sec 5 - Metrics primarily for receiver assessment. n-reordering goes here (n=3 predicts NewReno TCP unnecessary retransmissions). Potentially a modified version of the reordering density metric proposed. Matt noted that we should be discerning when adding metrics; there should be a good reason for adding another. A question from the group asked if reordering distance referred to a distribution or average. Al noted that it depends on whether you are talking about singleton metrics or statistics. The questioner clarified that he was assuming there are multiple samples. Stanslav mentioned mentioned that you can have a complete distribution if the number of values presented is small, and can then generate statistics. Al presented his idea of a potential fix to make n-reordering match the 4.2.3 singleton definition is to change "all" to "ANY" in Definition 1. The intent is to identify all packets that participate in a reordering event. [Chair note: We believe Al is looking to change the statement from universally quantified to existentially quantified. The proposed fix conveys this idea, but isn't correct mathematically (it has the same semantics as the original statement).] Greg Woodhouse asked if the measure is trying to quantify amount of effort (resources steps, memory) to restore order? Is it a hamming distance? Al Morton replied no, it is trying to quantify amount of buffer needed to restore order. Many metrics do not consider loss. Stanislav said that n-reordering was measuring the quantity of packets that are as good as lost for natural application behavior. Dave McDysan asked if the VoIP example dropped from Stanislav's draft? Stanislav answered that it was not intended to be dropped, will look at this again. Al Morton noted that what is important for VoIP is the time measure of reordering (a packet that arrives too late is as good as lost). In addition, VoIP jitter absorption buffers can reset. 5. IPPM Reporting MIB -- Jessie Jewitt http://www.ietf.org/internet-drafts/draf t-ietf-ippm-reporting-mib-02.txt See slides for detailed changes. Lots of editing for simplification, presentation, and to correct errors found when running SMILINT. They have a working prototype SNMP agent. TypeP was changed from an octet string to snmpadminstring for readability. Added scalars to control what happens when a log is full; suspend, resume and wrap the log were defined based on previous CMIP experience. Comments are solicited. A number of changes were made to the tables, based on implementation and working group suggestions. See slides. A couple of open issues were noted. First, is there a need for a count parameter for TypeP now that it is text instead of binary? Second, is there a problem with spaces in TypePaddress? Perhaps it too needs a count parameter. There were no immediate comments on these issues. Andy Bierman had some comments on the MIB, which he will send in detail to the list. In the meeting, Andy noted that some of the changes were not consistent across ippm OwnerString, SnmpAdminString. There is also still overlap with the control provided by the owners and shared-owners table and VACM. [A standard SNMPv3 access control method.] Emile noted that in the module compliance section this control is not mandatory if VACM is used; Andy did not feel that would be enough to get approved. The recommendation is to remove this control. Andy noted that the MIB has been cleaned up substantially; and that this is a very heavyweight (from an implementation resource point of view) MIB. This is in the same category as RMON probes with a dedicated processor and significant storage. The limit of the index on history (64K entries) seemed artificially low; Andy suggested that some case study thinking should be done to see how quickly this would wrap. Why wasn't it just made a 32 bit quantity? 6. Metrics Registry discussion --Emile Stephan http://www.ietf.org/internet-drafts/draf t-ietf-ippm-metrics-registry-02.txt Finally, and with only a few minutes remaining, Emile Stephan presented the changes to the metrics registry. See the slides for specific points. The biggest changes are with the tree (the "draft" subtree was removed), and instructions for draft authors that want to add a new metric to the registry were clarified. Emile thinks that the draft is stable, and is looking for a last call. He suggested that people review by mid-April, and then a last call should be issued. Andy Bierman noted that the registry looks good; it has been cleaned up. He also stated that RMON needs the registry. Matt said that a last-call will begin soon, and reiterated that the list will be moved to ietf.org (ippm@ietf.org) soon after this meeting. |