IETF56 Benchmarking Methodology WG Meeting Minutes

==================================================

The meeting was chaired by Kevin Dubray and Al Morton.  
Contributors to the meeting's minutes include Matt Zekauskas, Al Morton, and 
Kevin Dubray.  Special thanks to Matt for acting as meeting scribe.

About 75 people attended the BMWG session.  The original agenda:

0. Administration (Dubray, 15 min)
1. OSPF convergence I-D’s Last Call (White, 15 min)
2. Considerations Benchmarking Routing Protocol Network Convergence 
(White, 10 min)
3. IPsec Device Benchmarking Term. (Bustos, 20 min)
4. Traffic Control I-D update (Perser, 10 min)
5. Core Router Software Accelerated Life Testing (Poretsky, 30 min)
6. IGP Data Plane Convergence I-Ds. (Dubray, 10 min)

was approved without modification.


0. Administration
-----------------

Al Morton was welcomed as the new BMWG co-chair.

It was mentioned that the charter was revised to better reflect the 
lab-centric, vendor-independent nature of the BMWG's benchmarks.  
Milestones were brought current.  Interested parties were directed to the 
mailing list archive as the BMWG web page didn't reflect the Charter 
changes yet.

A status summary of BMWG Internet Drafts at the time of the meeting was 
presented as follows:

* AD/IESG Review:
    <draft-ietf-bmwg-conterm-04.txt>, revised to reflect IESG input.
    
<draft-ietf-bmwg-benchres-term-02.txt>, back in chairs’ domain.

* I-D Last Call:
    <draft-ietf-bmwg-fib-meth-01.txt>,  Call ended 3/14.
    <draft-ietf-bmwg-dsmterm-05.txt>, Call ends 3/25.
    
<draft-ietf-bmwg-ospfconv-term-02,.txt>, Clarifying input
    
<draft-ietf-bmwg-ospfconv-intraarea-03.txt>, Clarifying input
    
<draft-ietf-bmwg-ospfconv-applicability-01.txt> Clarifying input

* I-Ds:
     <draft-ietf-bmwg-mcastm-11.txt>, last call, task team, revised

* RFC Editor:
     <draft-ietf-bmwg-firewall-08.txt>, in RFC Editor queue.

* Expired BMWG I-Ds:
     <draft-ietf-bmwg-bgpbas-01.txt>, Pending term. progress
     
<draft-ietf-bmwg-benchres-method-00.txt> Pending term progress

* New Work proposals:
     Core Router Software Accelerated Life testing,
         
<draft-poretsky-routersalt-term-01.txt>
     Protection Switch I-Ds (To revise I-D before testing WG support),
         
<draft-kimura-protection-term-00.txt>
     IGP Convergence benchmarking,
         
<draft-poretsky-igp-convergence-term-00.txt>
         
<draft-poretsky-igp-convergence-meth-00.txt>
         
<draft-poretsky-igp-convergence-app-00.txt>


1.  Discussion on OSPF Convergence Benchmarking WG Last Call comments.

----------------------------------------
------------------------------
Discussion Goal: Last call is over, but consensus on outstanding issues 
isn't clear.  Identify outstanding issues.

Changes to Terminology Draft: Tree-shape discussion moves to the 
Applicability draft. Remove Duplicate Definitions for terms that are 
already defined, such as Shortest Path First (SPF). Changes in the 
Methodology Draft: The single "white box" test (section 8.3) becomes a 
supplement to (or validation of) the SPF convergence time black box test in 
section 8.4.

2.  Considerations in Benchmarking Routing Protocol Network 
Convergence.

----------------------------------------
---------------------------------
Discussion goal: Test whether the BMWG the right home for this I-D.

Russ White presented an overview of this draft, what was in/out of scope, 
and what was left to do next. He described a future investigation of this 
convergence testing, where a network of 300 routers would be used to 
evaluate a DUT, then substitute the networks with emulators and see what 
results are different, if any.

The memo will continue development as an individual draft, and Russ asked 
for comments. The WG can decide where this should go when the I-D is more 
mature (ippm did not seem like an option, since scope is 
emulation/lab net).

Presentation slides can be found in the Proceedings (bmwg-2)


3. IPsec Device Benchmarking Terminology I-D.

---------------------------------------------
Discussion Goal: Present I-D introduction and discuss current issues.

Michele gave an overview of the draft in its initial version, 
including an animated, two-phase tunnel establishment.  
Presentation slides can be found in the Proceedings. (bmwg-1)

There was a comment to add tunnel disengagement time and rate to the I-D 
since it seemed complimentary to tunnel establishment. Discussion 
revealed that the usual tunnel deletion is by timeout, and the "delete" 
capability for tunnel participants is optional.


4. Benchmarking Network-layer Traffic Control Mechanisms Terminology I-D

----------------------------------------
--------------------------------
Discussion Goal: Communicate latest changes & test progress of current WG 
Last Call.

There are several comments so far in Last Call and a few changes to make to 
the current I-D (including reflecting loss in the jitter 
measurement).  There will be another last call once the revised I-D is 
issued.

5. Terminology for Benchmark Core Router Software Accelerated Life 
Testing I-D

----------------------------------------
--------------------------------------
Discussion Goal: Version 00 of this draft seemed to present more 
functional testing than benchmarks - I-D fit to BMWG was questioned. This 
draft presumes better WG fit; did the principals succeed?

Scott Poretsky described the deltas that went into the draft since it was 
last proposed in Atlanta.  For details, see slides in the 
Proceedings. (bmwg-3)

Scott asked whether the I-D was ready to move forward to the ADs. The 
chairs advised that first we should see if the working group supports 
adding this I-D to the BMWG charter as a new deliverable.

There were concerns about whether the ensuing benchmarks could be 
specified with enough methodological detail so as to produce a uniform 
yardstick that would yield repeatable results. Scott indicated that he 
didn't believe detail regarding the associated methodologies would be an 
issue; moreover, he indicated that the benchmarks would be a good tool for 
multiple vendor comparisons.

There appeared to be interest from the attendees to test acceptance on the 
list of this I-D as a new BMWG project; the chairs indicated they would 
test acceptance of this proposed BMWG deliverable on the list.

IGP Data plane convergence benchmark I-Ds.

------------------------------------------
Discussion Goal: Regarding "convergence" benchmarking, the WG had 
restricted scoping of initial convergence benchmarks to the control 
plane.  However, the IGP convergence I-Ds were originally offered as a 
replacement to the OSPF convergence I-Ds. Should they replace the OSPF 
convergence IDs?  If not, is it time for the BMWG to consider explicit data 
plane convergence?  If yes, what is the IGP drafts' relationship to the 
OSPF convergence I-Ds: replacement on companion?

Kevin reminded the group that due to the complexity of 
characterizing convergence, the WG chose to take a stepwise approach in 
specifying related benchmarks: demonstrate progress in the control plane, 
then move onto other areas.

The group appeared non-committal as to whether the BMWG has achieved 
critical mass in the control plane convergence deliverables.  That said, 
there appeared to be a stronger opinion that we should start the data 
plane work as that is where practical meaning was to be found.

It was stated that IGP convergence seemed complementary to the current OSPF 
control plane convergence I-Ds.

There was talk of producing a master convergence document which sought to 
unify all that the individual I-Ds were prescribing.  Kevin recounted that 
the group considered producing a single work when initially 
discussing the convergence work.  It was thought, at the time, smaller 
deliverables were the way to go.  He offered that a single, unified 
document was not undesirable, but it may be prudent to wait until we have 
momentum and experience with the individual efforts from which to draw.

The chairs indicated that the would test acceptance of these proposed BMWG 
deliverables on the list.