|
<![CDATA[
Benchmarking Methodology WG (bmwg)
TUESDAY, August 2 at 1030-1230
==============================
CHAIRS: Kevin Dubray <kdubray@juniper.net>
Al Morton <acmorton@att.com>
Approximately 25 people attended the BMWG session at the
63rd IETF.
Matt Zekauskas volunteered to take raw meeting notes. [ed.:
Thanks, again(!), Matt!]
Agenda offered as:
0. Agenda
1. WG Status and Liaison notes (Chairs, 20 min)
2. Milestones Status (Chairs, 5 min)
3. Resource Reservation Terminology I-Ds. (Korn, 15 min)
4. IPsec Benchmarking Terms (& intro to methods) (Kaeo, 15 min)
5. Core Router Accelerated Life Testing I-Ds (Poretsky, 15 min)
6. Revised Charter Text (Chairs, 15 min)
7. New Work Proposals
0. Agenda
The Agenda was approved as proposed by the chairs. (The
chairs' presentation which contains the agenda and work
status report can be found in the proceedings; follow the
"Agenda/Status/Charter" link.)
The chairs reserved the right to move some agenda items around
in the discussion sequence.
1. WG Status and Liaison notes
The chairs presented the WG activity as a function of I-D
activity. (A detailed accounting can be found in the Chairs'
slides, slides 3, 4, & 5, in the Proceedings.)
Regarding some specific I-Ds:
* Network-layer Traffic Control Mechanisms Terms,
<draft-ietf-bmwg-dsmterm-11.txt> Working Group Last Call (WGLC)
requested, as S. Poretsky believes all nits are addressed.
Scott looking for help to start the methodology document.
* Hash & Stuffing: Overlooked Factors in Network Device
Benchmarking, <draft-ietf-bmwg-hash-stuffing-03.txt>, editors
requesting WGLC.
* Benchmarking IGP Data Plane Convergence benchmark I-Ds,
<draft-ietf-bmwg-igp-dataplane-conv-term-07.txt>,
<draft-ietf-bmwg-igp-dataplane-conv-meth-07.txt>,
<draft-ietf-bmwg-igp-dataplane-conv-app-07.txt>,
While WGLC yield no direct commentary to list, it did
yield one supporting comment from a methodology user,
and a comment on Consistency of THPT/FwdRate from Timmons Player.
There was considerable discussion of this off-list,
but the principals seem to be closing in on satisfactory text.
(Chairs remarked about the need to have discussions like this
on the BMWG mailing list.)
The BGP convergence methodology draft and resource reservation
methodology draft were cited as expired; the former has been
rumored to be started and the latter will get started upon
approval of its terminology peer.
Since the group met last, 4 BMWG I-Ds were promoted to
RFCs: RFCs 4061, 4062, 4063 (OSPF convergence) and RFC
4098 (on BGP convergence terminology).
* 802.11T Liaison
The chairs noted that a detailed report was posted (on 1 Aug 05)
to reflector from Tom Alexander on 802.11T WG. Of particular
note, a couple of proposals have been approved; these are
hoped to be integrated into a draft.
3. Resource Reservation Terminology I-Ds.
Andras Korn was present to discuss the latest version of the
"Benchmarking Terminology for Resource Reservation Capable
Routers," <draft-ietf-bmwg-benchres-term-06.txt>. Andras'
slides, can be found with the "Terms for Resource Res" link
in the Proceedings.
Andras described the latest changes in two classes:
editorial and technically significant. Most editorial changes
were to improve wording. The technical modifications brought
in the concept of loading (router and traffic), while expanding
the notions of loss (and loss-free conditions). Traffic units
were rethought.
A new section, 3.5, was added to better address load and
scalability limits. The principals tried to clarify the complex
boundary associated with load conditions and loss-free states.
The notion of "reasonable time" is a difficult concept to define,
but "reasonable time" is very important - especially
in the methodological considerations.
Andras cited this work has been going on for over 5 years; directed
input from group can help bring this phase to closure.
Al Morton cautioned that when work on the methodology starts,
use care that "reasonable time" doesn't become an acceptance
criteria standard. The notion of "waiting time" might be
set on the long side, record time-to-response, etc. Andras
countered while we don't determine acceptance standards, the
notion of correct behavior is crucial. Eliot Gerber noted
assessing "loss-free" conditions is important and questioned
its uniqueness to this effort.
Al added that given the significant changes, another WGLC will
be conducted after the summer holiday period.
4. Terms (and Methods) for Benchmarking IPsec Devices (T. Van Herck)
Slides by Tim Van Herck, Merike Kaeo's, and M. Bustos can be found in the
Proceedings under the "IPsec Terms & Method" link.
A driving motivation was stated to be defining tests that identify
implementation limits versus, say, interoperability. A methodology
doc was in the works, but input generated by the WGLC on the IPsec
terminology document demonstrated that reconsideration of the
tunnel definitions was warranted. In addition, modifications
to the terminology document was made to better address IPv6 and
enhance scope beyond gateway to some host testing.
Merike explained that while addressing the IPv6 definitions,
it became necessary to accommodate manual keying. Subsequently,
it was thought to rethink the tunnel terminology, too. Other
support (transport & tunnel modes) were added, too.
An informal poll showed that about 6 attendees read the
current I-D.
The editors feels that Phase 1 rekey frame loss could be supported,
but feel that it's not really needed. Similarly, support
for IKEv2 isn't a priority - v1 & v2 are substantially different
and would cause a substantial rewrite to the term doc.
Moreover, because v2 is a not backward compatible with v1 and
v2 addresses different application classes, it may be better
to address v2 in a separate, follow-on benchmarking document.
Support for fragmentation will be added - probably limited to
IPv4, but may need v6, too. Do we need a reassembly
metric, too? Reporting requirements were tightened to account
for Xauth/Modcfg. Traffic measurement units have been
respecified as frames - some discussion of this vis a vis
framesizes might be necessary. Again, lot's of tunnel terminology
being reworked.
Merike asked whether it is worth measuring "time to first
packet" - the time for tunnel setup plus propagation
delay to get cleartext packet from ingress port to
encrypted, egress port?
Merike reports the 00-version of the methodology I-D is
still solidifying. She is expecting a first draft in several weeks,
and hoping for some substantial feedback from the WG.
The editors request that the WG give input on target test topologies,
(currently 4 defined: single DUT, failover, IPsec aware and
unaware scenario), test set up, or otherwise cite improvements
or areas of omission.
Merike asked the chairs whether the editors could submit
the term and methodology docs for WGLC simultaneously in
a couple of months. Mr. Morton believed that was doable.
5. Core Router Accelerated Life Testing I-Ds.
Terminology & Methodology I-Ds revised. The Methodology has been split
into 3 drafts (general, EBGP, & security). Editor wishes to test
readiness
for WGLC on terms and general methods drafts.(S.Poretsky, in absentia)
Specifically, the following I-Ds, were addressed:
draft-ietf-bmwg-acc-bench-term-06.txt
draft-ietf-bmwg-acc-bench-meth-03.txt
draft-ietf-bmwg-acc-bench-meth-ebgp-00.txt
draft-ietf-bmwg-acc-bench-meth-opsec-00.txt
Slides for this presntation can be found in the BMWG proceedings
under the title "Several Drafts".
Al presented this topic on Scott's behalf, since he was unable to attend
at the last minute. It was clarified that the discussion at the last
meeting resulted
in a general recommendation to split the methodology draft into several
functional
areas (and not just a recommendation from the co-chairs). With many
first-time
BMWG attendees present, no one had read these drafts except the chairs.
Scott's slide described the two new drafts. The Operational EBGP
methodology contains
procedures for many different scenarios, and the Operational Security
draft has had
some input from the IETF opsec WG.
Al had one specific comment on the new ebgp methodology draft:
In several places, the Results section gives "expected" levels of packet
loss
and other expectations that may be that may constitute acceptance criteria
(as currently worded). It would be best to re-word this section to
simply identify
the key performance characteristics, adding the performance levels that
some
devices under test will achieve (zero packet loss). For example,
in Section 4.2, Results:
OLD
It is expected that there will be zero packet loss as the DUT
learns the new routes. Other DUT operation should be stable
without session loss or sustained packet loss.
NEW
Key benchmarks will be packet loss and session loss as the DUT
learns the new routes. Some DUT will achieve zero packet loss and
stable operation without session loss.
Also, it would be good to see some evidence that these benchmarks can be
used to make meaningful vendor comparisons, through example results
shared with
the working group. It's not mandatory to do this, but those who have doubts
may find this evidence compelling.
6. Revised Charter Text - Definition of a Benchmark (Chairs)
Al introduced the topic and covered several slides giving a proposed
benchmark definition, its attributes and exclusions. There was discussion
and feedback on several points.
Regarding the required attributes of a benchmark, Bert Wijnen asked if
specific coverage limits could be stated such that a reasonable scope
would be established, and without limits proposals for
electrical power outlets could appear! Al responded that the focus was
on IETF protocols and Kevin added that was the original intent, but that
this expanded to systems that use IETF protocols. The most recent questions
of BMWG scope were connected with benchmarks for 802.11 technologies. The
current charter says "inter-networking technologies and services", and
Kevin pointed out that wording is vague. The was agreement to tighten
the wording to help decide whether to accept or fend-off future work
proposals,
but the challenge is to develop the new wording.
Conformance tests were identified as excluded. Bert voiced his opinion
that
the ban was an IETF community position (rather than something the IESG
decided).
Kevin added that there has been pressure to include this sort of testing in
BMWG (if not there, where?), but Brian Carpenter's message to wgchairs list
on conformance testing ("...the IETF doesn't measure or certify conformance
or inter-operability, because that might create legal liability...")
makes the
IETF boundaries clear.
Acceptance tests are another exclusion. Andras Korn pointed out that it
is difficult to avoid specifying some minimum criteria for valid
benchmarking
in the resource reservation benchmarks. Al responded that it would be
appropriate
to specify some prerequisites for measuring benchmarks, and that it
would be up
to the testers to determine if the prerequisites are met. A methodology
could specify that devices must claim conformance with RFCxxxx, for example.
Andras pointed out that sometimes a protocol spec. does not include all
the specifications needed for testing (although it should). and additional
specifications may be needed to provide a complete methodology.
Tim Van Herck asked what org. IETF looks to for conformance testing,
since we cannot do it ourselves? Kevin responded that this is kind of a
catch-22,
and that there have been some correctness criteria specified in earlier
BMWG RFCs. We don't want good performance from a broken implementation.
Bert added that IETF should be writing protocol specifications that are
complete
and can be verified. IETF does not take a stance on vendors that lie about
conformance. Although vendor's customers would like a complete
certification
process (like CableLabs'), the change would require an
IETF-plenary-level discussion.
It would be good if non-compliant implementations produced poor benchmark
performance, but the results might favor a non-compliant implementation,
as well.
Merike Kaeo suggested that the methodologies should strive to make it
difficult to cheat. Al closed the discussion, saying that a more
fleshed-out
version of the definition (benefiting from the discussion today)
will be posted to the list.
2. Milestone Status
[ed. This agenda item was discussed out of sequence to the
proposed agenda.]
The Co-chairs summarized the progress toward meeting WG milestones in
three categories: Past Due, On-Track, and New. There was clear progress
on most of
the seven Past Due milestones, but the FIB and BGP methodologies were
exceptions.
Three milestones that are still on-track need attention to progress as
planned,
and we will need to add 2 new milestones to cover the new Accelerated
Stress Methodologies on EBGP and Operational Security. We need to revise
the milestones and obtain AD-approval.
7. New Work Proposals
* draft-poretsky-protection-term-00.txt
* draft-poretsky-mpls-protection-meth-04.txt
Sub-IP Protection terms and a method for MPLS have been harmonized into
a single
proposal now. There was good support for this work at our Nov 2004 meeting,
but at this meeting, no one (other than the chairs) had read the drafts.
David Kessens indicated his concern about taking up new work without the
support of the WG, although this particular meeting was lightly attended
by the WG's most active participants. David agreed that the work could be
proposed to the list, but that he was raising the bar to accept new work.
He wants to see more than just interest, people must be willing to
contribute
and people must be willing to say that they need this work done. If the
work is done by two people, then it's not IETF work and should be published
by the individuals.
8. WG Chairs' Summary and Conclusions
There are several WG Last Calls to Launch after the meeting, including the
benchres, IGP-dataplane, and hash-and-stuffing drafts. The IPsec
authors are
looking for people to review the new methodology, and hope to start another
last call in a few months. We will also float the proposal for new work
on the list, with a high bar for acceptance. If list members are part of
the WG, then participation isn't optional! We must have participation
to move topics forward, and we encourage everyone to find a way they
can contribute to the work at hand. Otherwise, BMWG will work in other
areas with clear sponsorship and participation.
]]>
|
