Benchmarking Methodology WG (bmwg)


Tuesday, July 15, 2003, 1300-1400 and 1415-1515
=======================================


CHAIRS Kevin Dubray <kdubray@juniper.net>
          Al Morton <acmorton@att.com>


Reported by Al Morton, based on the information generously compiled by Sue 
Hares as official note-taker.


About 66 people attended one or more of the BMWG sessions.


1.   Working Group Status (Morton)
Status of BMWG I-Ds at close of IETF-57


AD/IESG Review
<draft-ietf-bmwg-conterm-05.txt>, revised to reflect IESG input, WG LC 
soon.


I-D Last Call
<draft-ietf-bmwg-fib-meth-01.txt>,  Call ended 3/14.
<draft-ietf-bmwg-dsmterm-07.txt>, Call ended 7/15 with comment.
<draft-ietf-bmwg-mcastm-13.txt>, Call ended 7/5, to ADs soon

<draft-ietf-bmwg-ospfconv-term-05.txt>, Call ended 7/1 with 
comments.

<draft-ietf-bmwg-ospfconv-intraarea-04.txt>, Call ended 7/1

<draft-ietf-bmwg-ospfconv-applicability-03.txt> Call ended 7/1


I-Ds
<draft-ietf-bmwg-dsmmeth-00.txt>, coming soon

<draft-ietf-bmwg-ipsec-term-01.txt>(draft 07/2003, need readers)

<draft-ietf-bmwg-benchres-term-03.txt>, back in WG
<draft-ietf-bmwg-acc-bench-term-00.txt> New 06/2003

<draft-ietf-bmwg-igp-dataplane-conv-term-00.txt> New 06/2003

<draft-ietf-bmwg-igp-dataplane-conv-meth-00.txt> New 06/2003

<draft-ietf-bmwg-igp-dataplane-conv-app-00.txt> New 06/2003



2.   Discussion on OSPF Convergence Benchmarking WG Last Call 
comments. Last call is over, but a few comments need discussion.  
Identification of remaining issues.  (Morton)



http//www.ietf.org/internet-drafts/draft
-ietf-bmwg-ospfconv-term-04.txt

http//www.ietf.org/internet-drafts/draft
-ietf-bmwg-ospfconv-intraarea-05.txt

http//www.ietf.org/internet-drafts/draft
-ietf-bmwg-ospfconv-applicability-03.txt


There was one remaining comment from the Last Call ending July 1, asking to 
remove definitions of "Internal Measurements" and "External 
Measurements". Following a short discussion, the commentor 
(S.Proetsky) agreed that the definitions should remain, but that the terms 
most frequently used in BMWG drafts and discussion are "White Box" and 
"Black Box" measurements. The term names should be changed to add the 
"Boxes" as synonyms and any resulting changes to the definition text. This 
potential resolution will be discussed with authors and on the list.



3.  Benchmarking Network-layer Traffic Control Mechanisms 
Terminology -- (J. Perser et al.)


    
http//www.ietf.org/internet-drafts/dra
ft-ietf-bmwg-dsmterm-07.txt


Jerry Perser communicated the latest changes and offered one last chance to 
comment during WG Last Call. There was one comment on the "Channel 
Capacity" term and its definition from Tony DeLaRosa. This comment was 
resolved during a face-face meeting later in the week.


    Also, Future Draft on Benchmarking Traffic Control Mechanisms 
Methodology --


Scott Poretsky gave a brief introduction and identified areas for 
comment and contribution during development. The purpose statement is to 
provide methodologies to benchmark devices "capable of delivering the 
specific packet forwarding treatment indicated by the DS field value" 
[RFC2474].  The Benchmark is a Device’s ability to meet configured PHBs 
(the Expected Vector), not on specific QoS mechanisms. This makes it 
possible to compare performance of routers implementing different 
mechanisms. A short discussion revealed that measurement time scale will be 
important for EF evaluation.


4.  Terminology for Benchmarking BGP convergence in the Control Plane

http//www.ietf.cnri.reston.va.us/interne
t-drafts/draft-ietf-bmwg-conterm-05.txt


This draft was recently revised, and there will be a short WG Last Call 
before returning it to the ADs.



5.  IPsec Device Benchmarking Terminology I-D --  (M. Kaeo, et al.)


    
http//www.ietf.org/internet-drafts/dra
ft-ietf-bmwg-ipsec-term-01.txt


The text of IPsec Terminology was completed in this version (01). Merike 
asked BMWG if there were any terms missing, any problems with 
Definitions, and if the group believed this draft was ready for WG Last 
Call. There were several readers: one offering an endorsement, and two 
questions on the scope (NAT devices and IKEv2 are out of scope, however 
NAT-Transversal is included in the scope of this effort). The current 
draft defined IMIX (Internet Mix for traffic synthesis), but did not 
include the reference that Michele Bustos provided on the list. There was a 
comment suggesting that this is a good definition to include, with the 
caveat that no one mix of packet sizes represents Internet usage. There 
will be a revision to make these points clear, and then a WG Last Call.



6.  IGP Data plane convergence benchmark I-Ds --   (S.Poretsky)



http//www.ietf.org/internet-drafts/draft
-ietf-bmwg-igp-dataplane-conv-term-00.txt

http//www.ietf.org/internet-drafts/draft
-ietf-bmwg-igp-dataplane-conv-meth-00.txt

http//www.ietf.org/internet-drafts/draft
-poretsky-igp-convergence-app-00.txt


Scott Poretsky gave a brief introduction of the new work item. All three 
drafts are progressing with comments. There were several bmwg-list 
suggestions for terminology changes, and they were illustrated in the 
viewgraphs. Discussion of the methodology identified the need for a note to 
cover forwarding rate change after convergence, and another suggested 
specific reporting formats that remove the effect of delay timers. There was 
also a discussion of measurement sampling rate and its affect on the 
accuracy of loss-based convergence time measurements when 
convergence approaches the sampling interval.



2nd Session
Tuesday, July 15, 2003, 1415-1515
=======================================


7.  Resource Reservation Terminology benchmark I-Ds --  (Fehér Gábor Gume)



http//www.ietf.cnri.reston.va.us/interne
t-drafts/draft-ietf-bmwg-benchres-term-03.txt


Description of changes based on AD review, primarily to remove Diffserv 
material, sync-up with current NSIS drafts, and remove non-standard 
examples (e.g., Boomerang).There were still a few places where comments 
could be more thoroughly adopted, and the draft will be revised again 
shortly. Gábor made an appeal for more readership and review on 
BMWG-list.



8.  Terminology for Benchmarking Core Router Accelerated Life Testing. (S. 
Poretsky et al.).



http//www.ietf.cnri.reston.va.us/interne
t-drafts/draft-ietf-bmwg-acc-bench-term-00.txt


Comments asked for examples of Management Plane Failures, and 
identified problems with allowing complete flexibility in test 
configurations (results comparisons will likely be confounded by 
differences). The level of readership and review is not yet clear for this 
draft.



9.  Proposed Milestone Revisions -- (Morton)


The review of milestones revealed a growing list, and a need to achieve 
"Done" on a few more before adding new work. Editors and list members 
should comment on this proposal (on the list).


   Done  First I-D on IPsec Device Benchmarking Terminology
   Aug 03  Methodology for IP Multicast Benchmarking to AD Review.
   Nov 03  Resource Reservation Benchmarking Terminology to AD Review
   Aug 03  Net Traffic Control Benchmarking Terminology to AD Review
   Dec 03  Methodology for FIB related Router Performance 
Benchmarking to AD review.
   Dec 03  EGP Convergence Benchmarking Terminology to AD Review
   Mar 04  Resource Reservation Benchmarking Methodology to AD Review
   Dec 03  Basic BGP Convergence Benchmarking Methodology to AD Review.
   Dec 03  Net Traffic Control Benchmarking Methodology to AD Review.
   Dec 03  IPsec Device Benchmarking Terminology to AD Review


   Dec 03   AD review on IGP/Data-Plane Terminology I-D
   Mar 04   AD review on IGP/Data-Plane Methodology and 
Applicability I-Ds


   Nov 03   AD review on Router Accelerated Test Terminology I-D
   Jul 04   AD review on Router Accelerated Test Methodology and 
Applicability I-Ds


10. New work proposal Automatic Protection Switching Benchmark 
Terminology (T.Kimura, J.Perser)
 
http//www.ietf.org/internet-drafts/draft
-kimura-protection-term-01.txt


    Related Individual Draft on MPLS Protection Benchmarking 
Methodology (S.Poretsky, et al.)
 
http//www.ietf.org/internet-drafts/draft
-poretsky-mpls-protection-meth-00.txt


The Goal has been articulated as follows:


    The objective of this effort is to produce a terminology and 
methodology set of drafts that specifies the performance 
benchmarking sub-IP layer resiliency and protection technologies. There is a 
common terminology draft and multiple methodology drafts for the 
technologies.  The methodology drafts will include (but not limited to) 
Automatic Protection Switching (APS) for SONET/SDH, Fast Reroute for 
Multi-Protocol Label Switching (MPLS), and Resilient Packet Ring (RPR) 
standardized in IEEE.


Discussion revealed that the previous direction to have a common 
terminology draft for this work will be stretched by the plethora of Layer 2 
terms, and so this issue must be worked further to see if a common 
terminology is possible. The best way forward may be to link the 
technology-specific terms to the appropriate IP layer terms. Recovery time is 
key concept: accurate measurement takes all 5 recovery types into 
account: Lost packets, Induced delays, Duplicate packets, 
Out-of-order packets, and Errored packets. Clearly, measuring packet loss 
alone to assess Recovery Time will not be sufficient in some cases. The 
general assumption here is that the protection recovery times will be 
sufficiently fast to avoid triggering IGP reconvergence, in response to 
Craig White's inquiry on this interaction. Craig also commented that other 
transport components interact with these protection mechanisms, and Tony 
DeLaRosa added that a System Under Test may be needed to accommodate this 
point. Matt Mathis suggested using general terms for the paths, such as 
primary and secondary (and tertiary), and that we should invent some term 
for the triggered failure, like "induced failure." Also, a generic IP 
Protection Methodology may be useful to benchmark unspecified 
protection mechanisms. The discussion concluded that the proposal 
requires more work before review by the WG.