2.4.3 Benchmarking Methodology (bmwg)

NOTE: This charter is a snapshot of the 50th IETF Meeting in Minneapolis, Minnesota. It may now be out-of-date. Last Modified: 14-Mar-01

Chair(s):

Kevin Dubray <kdubray@juniper.net>

Operations and Management Area Director(s):

Randy Bush <randy@psg.com>
Bert Wijnen <bwijnen@lucent.com>

Operations and Management Area Advisor:

Randy Bush <randy@psg.com>

Mailing Lists:

General Discussion:bmwg@ietf.org
To Subscribe: bmwg-request@ietf.org
In Body: subscribe your_email_address
Archive: ftp://ftp.ietf.org/ietf-mail-archive/bmwg/

Description of Working Group:

The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies.

Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results.
Because the demands of a class may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements.
An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation internetworking technologies.

Goals and Milestones:

Done

  

Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches.

Done

  

Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC.

Done

  

Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately.

Done

  

Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft.

Done

  

Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices.

Done

  

Submit initial draft of Benchmarking Methodology for LAN Switches.

Done

  

Submit Terminology for IP Multicast Benchmarking draft for AD Review.

Done

  

Submit Benchmarking Terminology for Firewall Performance for AD review

Done

  

Progress ATM benchmarking terminology draft to AD review.

Done

  

Submit Benchmarking Methodology for LAN Switching Devices draft for AD review.

Jul 00

  

Submit first draft of Firewall Benchmarking Methodology.

Done

  

First Draft of Terminology for FIB related Router Performance Benchmarking.

Done

  

First Draft of Router Benchmarking Framework

Jul 00

  

Methodology for FIB related Router Performance Benchmarking to AD review.

Sep 00

  

Methodology for IP Multicast Benchmarking to AD Review.

Sep 00

  

Methodology for ATM Benchmarking for AD review.

Sep 00

  

Progress Frame Relay benchmarking terminology draft to AD review.

Nov 00

  

Terminology for ATM ABR Benchmarking for AD review.

Jan 01

  

Firewall Benchmarking Methodology to AD Review

Mar 01

  

First Draft of Methodology for FIB related Router Performance Benchmarking.

Mar 01

  

Router Benchmarking Framework to AD review.

Jul 01

  

Terminology for FIB related Router Performance Benchmarking to AD review.

Internet-Drafts:
Request For Comments:

RFC

Status

Title

RFC1242

 

Benchmarking Terminology for Network Interconnection Devices

RFC2285

 

Benchmarking Terminology for LAN Switching Devices

RFC2432

 

Terminology for IP Multicast Benchmarking

RFC2544

 

Benchmarking Methodology for Network Interconnect Devices

RFC2647

 

Benchmarking Terminology for Firewall Performance

RFC2761

 

Terminology for ATM Benchmarking

RFC2889

 

Benchmarking Methodology for LAN Switching Devices

Current Meeting Report

Benchmarking Methodology WG Minutes

WG Chair: Kevin Dubray
Minutes reported by Kevin Dubray.

The BMWG met at the 50th IETF in Minneapolis, MN, on Monday, March 19, 2001.

The proposed agenda:

1. Administration
2. <draft-berkowitz-bgpcon-00.txt>
3. <draft-ietf-bmwg-dsmterm-00.txt>

was approved.

1. Administration.
------------------

Activity over the last period was summarized as:

The I-Ds: <draft-ietf-bmwg-atm-term-abr-02.txt>, <draft-ietf-bmwg-fr-term-05.txt>
now reflect IESG inputs and should be on their way to RFC editor.
[Chair's note: Currently, there is some confusion over edit versions of these drafts. This should be resolved soon.]

The I-D <draft-ietf-bmwg-atm-method-03.txt> passed a WG last call and should be in the ADs' hands for Info RFC consideration.

The I-Ds: <draft-ietf-bmwg-mcastm-07.txt>, <draft-ietf-bmwg-fib-term-01.txt>
were revised but missed the I-D cutoff. They will be resubmitted after the IETF meeting.

2. <draft-berkowitz-bgpcon-00.txt>
----------------------------------

Sue Hares and Howard Berkowitz were present to give insight on their draft about measuring BGP convergence. It was stated that a second draft was produced, but it was unable to be submitted before the Internet Draft cut off time. The presentation and discussion, however, surrounded that second draft. Slides for the presentation can be found at:
http://www.nexthop.com/standards/presentations/bmwg.ppt

The presenters reinforced the notion that the scope of this work was very constrained. Moreover, they further explained that there were many topics of convergence that would be left to future work. Examples of some of these future convergence topics include BGP and IGP interaction, flapping, and policy. While there were some concerns of worm can opening, there appeared to be attendee interest in this work. The presenters were requested to submit the revised draft to the Internet-Draft secretariat, after which a proposal to undertake this new initiative would be made to the working group at large.

2. <draft-ietf-bmwg-dsmterm-00.txt>
-----------------------------------

The authors of this first draft on terminology for benchmarking network layer traffic control mechanisms were on hand to summarize the status of the draft and to field questions. Essentially, it was stated that this first draft was a work in progress and, as such, it was not near completion. The bulk of the comments offered counsel to align this new work with previous BMWG work, wherever possible.

It was noted that some drafts had expired, specifically the Router benchmarking framework document. The chair ask the group to consider the continuation of this dormant work items.

Lastly, the chair invited the group to consider and offer comments on the individually submitted draft regarding the standardization of metrics in the IETF. The draft ID is:
draft-bradner-metricstest-00.txt.

The goals for next period are:

* Work to bring the firewall, multicast, and FIB I-Ds to a Working Group Last Call.

* Progress the traffic control mechanisms draft and Resource Reservation drafts.

* Assess the proposed new work item on BGP convergence benchmarking.

Slides

None received.