NOTE: This charter is a snapshot of the 49th IETF Meeting in San Diego, California. It may now be out-of-date. Last Modified: 30-Oct-00
Chair(s):
Kevin Dubray <kdubray@ironbridgenetworks.com>
Operations and Management Area Director(s):
Randy Bush <randy@psg.com>
Bert Wijnen <bwijnen@lucent.com>
Operations and Management Area Advisor:
Randy Bush <randy@psg.com>
Mailing Lists:
General Discussion:bmwg@ironbridgenetworks.com
To Subscribe: bmwg-request@ironbridgenetworks.com
Archive: http://www.alvestrand.no/archives/bmwg/
Description of Working Group:
The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.
Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.
Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.
An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.
Goals and Milestones:
Done |
|
Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches. |
Done |
|
Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC. |
Done |
|
Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately. |
Done |
|
Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft. |
Done |
|
Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices. |
Done |
|
Submit initial draft of Benchmarking Methodology for LAN Switches. |
Done |
|
Submit Terminology for IP Multicast Benchmarking draft for AD Review. |
Done |
|
Submit Benchmarking Terminology for Firewall Performance for AD review |
Done |
|
Progress ATM benchmarking terminology draft to AD review. |
Done |
|
Submit Benchmarking Methodology for LAN Switching Devices draft for AD review. |
Jul 00 |
|
Submit first draft of Firewall Benchmarking Methodology. |
Done |
|
First Draft of Terminology for FIB related Router Performance Benchmarking. |
Done |
|
First Draft of Router Benchmarking Framework |
Jul 00 |
|
Methodology for FIB related Router Performance Benchmarking to AD review. |
Sep 00 |
|
Methodology for IP Multicast Benchmarking to AD Review. |
Sep 00 |
|
Methodology for ATM Benchmarking for AD review. |
Sep 00 |
|
Progress Frame Relay benchmarking terminology draft to AD review. |
Nov 00 |
|
Terminology for ATM ABR Benchmarking for AD review. |
Jan 01 |
|
Firewall Benchmarking Methodology to AD Review |
Mar 01 |
|
First Draft of Methodology for FIB related Router Performance Benchmarking. |
Mar 01 |
|
Router Benchmarking Framework to AD review. |
Jul 01 |
|
Terminology for FIB related Router Performance Benchmarking to AD review. |
Internet-Drafts:
· Terminology for Frame Relay Benchmarking
· Methodology for IP Multicast Benchmarking
· Methodology for ATM Benchmarking
· Terminology for ATM ABR Benchmarking
· Terminology for Forwarding Information Base (FIB) based Router Performance Benchmarking
Request For Comments:
RFC |
Status |
Title |
RFC1242 |
Benchmarking Terminology for Network Interconnection Devices | |
RFC2285 |
Benchmarking Terminology for LAN Switching Devices | |
RFC2432 |
Terminology for IP Multicast Benchmarking | |
RFC2544 |
Benchmarking Methodology for Network Interconnect Devices | |
RFC2647 |
Benchmarking Terminology for Firewall Performance | |
RFC2761 |
Terminology for ATM Benchmarking | |
RFC2889 |
Benchmarking Methodology for LAN Switching Devices |
Benchmarking Methodology WG Minutes
WG Chair: Kevin Dubray
Minutes reported by Kevin Dubray.
The BMWG met at the 49th IETF in San Diego, CA., on Monday,
December 11, 2000.
The proposed agenda:
1. Administration/Agenda tweaking
2. Terminology for Forwarding Information Base (FIB) based Router Performance Benchmarking.
<draft-ietf-bmwg-fib-term-00.txt>
3. Benchmarking DiffServ Mechanisms
was amended to include an update on the status of the Individually Submitted I-Ds on Resource Reservation benchmarking.
1. Administration.
Dubray announced that I-Ds <draft-ietf-bmwg-fr-term-05.txt> and <draft-ietf-bmwg-atm-term-abr-02.txt> came back from IESG review and that they were being modified with minor edits. I-D <draft-ietf-bmwg-atm-method-03.txt> was in WG Last Call. It was announced that the BMWG would undertake the new work item, DiffServ Mechanisms Benchmarking. The Multicast Benchmarking Methodology I-D had a few outstanding issues - it was asked that wording for contentious areas be offered so that the draft could be moved to last call.
2. Terminology for Forwarding Information Base (FIB) based Router Performance Benchmarking.
Guy Trotter led a discussion on <draft-ietf-bmwg-fib-term-00.txt>. It was thought that the notion of "fully meshed" was a methodological concern and should be addressed in the subsequent methodology document, as was specification of an aggregation heuristic. It was noted that this draft will target IPv4 exclusively; IPv6 could be addressed in a supplemental work. There was discussion regarding the desire to possibly include mechanisms to characterize the effect of prefix length on forwarding performance. A table of contents was suggested as a good thing towards improving the I-D; explicit identification of metrics from the supporting terminology was thought to be desirable, too. There was discussion on the attributes to consider in defining the term "FIB". A question was asked whether the author considered employing related MIBs in the benchmarks. It was thought that the question was interesting, but may be a more invasive (i.e., less than a black box) examination of the DUT. It was asked the meeting attendees pose their thoughts/comments/questions to the BMWG mailing list. Guy's presentation is appended to the Proceedings.
3. Benchmarking DiffServ Mechanisms
Jerry Perser presented some thoughts on issues and metrics in benchmarking DiffServ mechanisms. While the presentation addressed measurements of congested vs. uncongested DUTs, the bulk of the discussion revolved around the notion of measuring DUT queue size. Black box or white box test? One queue or N queues? The discussion wasn't conclusive; it was suggested that further discussion be taken to the mailing list. A question was asked regarding the effort's intent to derive metrics that would give insight into a device's ability to enforce contracts or SLAs. It was thought that this is a more end-to-end thing - however a DUT's characterization might be a component that provides insight into the "bigger picture." Jerry's presentation can be found in the Proceedings.
4. Resource Reservation Benchmarking
Krisztian Nemeth gave the group a short overview of the effort, its related models, and some preliminary findings. He then updated the group as to the history and status of the relevant individual I-Ds. Initially, <draft-feher-benchresres-00.txt> was issued and presented to the BMWG in Pittsburgh; there was some preliminary interest and the authors followed the Pittsburgh group's recommendation to break the draft into separate terminology and methodology documents. The I-Ds,
<draft-feher-bmwg-benchres-term-00.txt> and
<draft-feher-bmwg-benchres-method-00.txt>
were sent to the I-D secretariat, but only the methodology I-D was issued. Krisztian noted that responses to the work on the mailing list was light, but he again asked the BMWG to undertake this work. The chairperson asked Krisztian to check with the I-D secretariat and have the terminology draft issued. Once the I-D is issued, a WG Last Call on the New Work Proposal would be announced. Krisztian's presentation can be found in the Proceedings, too.
5. Goals for the Next Period.
5.1 Complete WG Last Call on the ATM methodology I-D;
5.2 Bring the multicast benchmarking terminology I-D to WG Last Call;
5.3 Advance the FIB terminology benchmarking draft.
5.4 Advance the Firewall methodology benchmarking draft.
5.5 Revisit the Router Benchmarking Framework document.
5.6 Issue the first WG I-D on DiffServ Mechanism Benchmarking Terminology in February.
5.7 Issue a WG Last Call on a New Work Proposal regarding Resource Reservation Benchmarking.
Terminology for FIB Based Router Performance Benchmarking
Metrics for Measuring Differentiated Service Performance
Benchmarking Terminology and Methodology for Routers Supporting Resource Reservation