2.2.2 Problem Statement (problem)

NOTE: This charter is a snapshot of the 56th IETF Meeting in San Francisco, California USA. It may now be out-of-date.

Last Modified: 2003-03-10

Chair(s):
Avri Doria <avri@acm.org>
Melinda Shore <mshore@cisco.com>
General Area Director(s):
Harald Alvestrand <harald@alvestrand.no>
General Area Advisor:
Harald Alvestrand <harald@alvestrand.no>
Mailing Lists:
General Discussion: problem-statement@alvestrand.no
To Subscribe: problem-statement-request@alvestrand.no
Archive: http://www.alvestrand.no/pipermail/problem-statement/
Description of Working Group:
Discussions during 2002 have revealed a significant number of thoughts about problems that exist with the way the IETF operates. In advance of trying to change the IETF procedures and rules to deal with these problems, the IETF should have a clear, agreed description of what problems we are trying to solve.

This group is charged with producing the document describing these problems. The analysis of the problem should seek out the root causes of the problems as well as the perceived derivative problems.

The intent is that the group will discuss issues on its mailing list, and that there will be an editing team to produce a clear concise problem statement on which the group has come to consensus and present to the IETF as a basis for an IETF consensus.

As a second work item, the group will also produce a proposal for a process to develop solutions to the problems identified by this working group.

It is not a part of this group's charter to propose solutions to the problems.

The work items will be reviewed in IESG plenary at the IETF.

Goals and Milestones:
Done  First I-D of problem statement issued
MAR 03  Problem statement reviewed at the IESG Plenary
MAR 03  First I-D of draft document describing the process by which the ietf will change its processes issued
MAY 03  Problem statement submitted for IESG review
JUL 03  Draft document describing the process by which the ietf will change its processes reviewed at the IESG Plenary
AUG 03  Draft document describing the process by which the ietf will change its processes submitted for IESG review
OCT 03  Re-charter or close working group
Internet-Drafts:
  • - draft-ietf-problem-issue-statement-00.txt
  • No Request For Comments

    Current Meeting Report

    Problem Statement BOF (problem)
    
    Friday, March 21 at 0900-1130
    ==============================
    
    CHAIR:	Melinda Shore <mshore@cisco.com>
    	Avri Doria <avri@apocalypse.org>
    
    Notes:	Spencer Dawkins <spencer@mcrs-labs.org>, Chris Allen
        <ChristopherA@AlacrityManagement.com> 
    
    Charter discussion - Chairs
    	Charter went through quickly with little controversy
    	work is to be analytic and descriptive, with emphasis on analytic
    	not pointing fingers
    	aggressive schedule - done in October
    	comments on document structure and contents both welcome
    	
    Current Status
    	Mailing list but no separate web page - do we need one?
    	1st draft of problem description document has been released
    	Have editors and editor team for both chartered documents
    
    Chris Allen: suggested addition to charter - document existing 
    procedures to keep what's effective
    
    Document Process Performance - Henning Schulzrinne
    
    Identified "IETF Goodput Measures": size, delay, throughput.  This was a 
    first attempt to figure out if this is worth looking at.
    Methodology: matched RFC names with draft-00 titles, looking for 
    "end-to-end latency."  Our data are extremely poor - no other 
    mechanism works.  The Internet Monthly Report doesn't contain all I-D 
    names and the IETF Announcement list only goes back to 1998 so history has 
    been lost.  Help is welcome on this.
    
    Average number of pages spiked a lot earlier than number of RFCs per year. It 
    hasn't changed since early 80s -- about 200 RFCs per year, about 25-30 
    pages per RFC.  Internet draft bandwidth has been on steady linear 
    increase since 1991.  2001 was down, 2002 was back up.  00-to-RFC delay has 
    grown from 6 months to about two years, with a steady increase from 1991 to 
    1998, and constant since then.  Harald Alvestrand asked if Henning had 
    tried plotting the graph using the year the work was started instead of the 
    year the work was ended.  Henning said he hadn't.  In response to a 
    question from Chris Allen Henning said that many measures actually fell off 
    somewhat in 2001, not just these.  Several people suggested 
    additional metrics and Henning responded that other statistics are also 
    interesting
    
    The RFC Delay Histogram clusters at two years, but has a long tail - four, 
    even five years.  Aaron Falk asked about individual submissions that 
    become working group document, and Henning responded that the stats were 
    based on title matching, not I-D name matching.
    
    The data are VERY noisy - formats highly variable, Internet Monthly 
    Report is incomplete, some RFCs were never announced.  The estimates are 
    likely low due to I-D renaming or high due to "blocking."
    
    Some suggestions: sample, don't census
    	Look at questions like "one rev per IETF meeting"
    	Waiting for other documents? author fixes? IESG queues?
    	Look at reasons, not at data
    Does this help agreement on root causes?
    
    
    Initial Analysis of IESG Review - Eric Rescola
    
    Overlaps Henning's analysis.  He also experienced low signal-noise ratio, 
    but felt his findings were basically correct.  The question being 
    addressed was how long do things take from last call to document 
    approval.
    
    What he found was:
    	mean = 147 days, median = 93 days - standard deviation is 149 days
    	300 days isn't unusual
    	Documents that are published as is are still slow
    	Areas differ a bunch in speed:
    		internet, Ops and transport are the fast ones
    		applications and security are the slow ones
    	and differ a bunch in output volume
    
    Churning takes time (60 days, volume of changes doesn't matter, length 
    doesn't matter.  The approval time is constant over study period.
    
    Dave Crocker said this work was similar to work Marshall and he did and 
    that there were some methodological problems with the treatment of the 
    data.  he said that he was not saying the math was wrong, but use of those 
    statistics in that environment is risky because the community of 
    professionals use normal data, but don't know how to process 
    log-normal data. There is a difference between the math world and the 
    social research world.  Eric disagreed about the appropriateness of the 
    tests and said that the methodology was fine.
    
    Bill Sommerfield said that as a WG chair he's frustrated.  In his 
    working group they get one bug report, they fix it, submit, get one more bug 
    report, fix it, etc.  Are there data on small changes?  Eric answered from 0 
    to 4 review cycles typically.
    
    Aaron Falk said that AD review isn't well documented and can be highly 
    variable, and Eric replied that it can be highly variable between ADs, as 
    well.
    
    
    Architectural and Process Inputs - James Kempf
    
    James's discussion was titled "Helping the Internet to age 
    gracefully." 
    
    Heard around IETF: "We don't do any architecture," with 
    architectural work hidden as "frameworks", etc.  Heard around 3GPP - "IETF 
    doesn't do architecture, they do philosophy."
    
    End-to-end is an example of traditional approach and it's helpful, but we 
    need more.  We must integrate new components with existing 
    components.  There are interdependencies across working groups, or even 
    across areas.  Catching them at IESG review time is too late to affect base 
    design.  Is this really system design, and not architecture?
    
    IETF work is a stove pipe, and assumes minimal 
    interdependencies - but the number isn't zero.  Can we ignore the 
    problem?  We're sending more drafts back due to conflicts, but missed 
    interdepencies still happen.  There's an increasing lack of 
    transparency and understandability in Internet Design.  It's becoming more 
    like the telephony world - baroque.
    
    Bob Hinden raised the issue of proposed standards and said that there are 
    risks in implementing PS, but they implement PS in products all the time.  
    James replied but the problem might be big - not a minor tweak, and Bob 
    pointed out that full standards are really rare.
    Bill Sommerfeld: too difficult to move past PS - there's no energy after doc 
    is stable.
    
    Henning said that he doen't disagree with slides, but needs case studies for 
    background, with concrete examples of last-minute/too-late fixes.
    James: this would be useful.
    
    	
    Working Group Participation and the "Stuckee Problem" - Ted Hardie
    
    The discussion is abstracted from and I-D that's 
    solutions-based.  The IETF working style of an Open working group on 
    mailing lists has produced successful results, but making a comment 
    doesn't mean you're taking responsibility for doing work.  It's 
    difficult to make anyone other than WG chairs and current authors 
    accountable.  The problem is how to track commitment without breaking 
    openness and shutting out outside review.
    
    Dave Crocker said that we have people who consume resources but don't help 
    make progress.  Ted said that may be true but that isn't the problem he's 
    looking at.  Dave replied that the working group isn't real if people 
    don't show up and do work, and that what Ted is saying is reasonable but 
    represents a different philosophical or strategic difference from what the 
    IETF ought to be.  
    
    Ted answered that working groups do vary over time, but stuff does fall out 
    the other end.  There may not be a constituency when it does, but that's not 
    what we are discussing here.  The problem is that we can't do anything that 
    has dependencies unless someone is actually responsible for those 
    dependencies.  Spencer Dawkins said that dependencies are worse for other 
    SDOs that depend on the IETF to produce a standard. 
    
    How can working group chairs responsibly say "you can count on us?  Bob 
    Hinden said that if there's no energy, the work shouldn't be done, and Ted 
    replied that if another working group is depending on the work that's not 
    being done, the second working group either fails or takes on all the 
    work.
    
    
    Problem Resolution status - Margaret Wasserman
    
    This is the second document in working group charter.  There are clearly 
    issues about changing the IETF.  Where do we go from here?  The IETF is 
    unique, but these aren't unique problems.  Good times hide problems, bad 
    times hide successes, and we need a consistent focus on operational 
    improvement.
    
    Margaret listed Harald's core values from IETF 55 plenary.  Things that 
    aren't core values are all on the table for changes.  The current 
    environment is highly emotional but we're committed to improvement.  The 
    IETF is a fast-moving target - we can't stand still while we measure and 
    decide how to move forward.
    
    Major process options:
    	Evolution vs revolution ("nothing left to lose"?)
    	Granular vs monolithic (can problems be separated?)
    	Immediate versus ongoing
    	Grassroots vs top-down
    
    What are the right mechanisms to use, moving forward?  Should we update 
    existing process documents?  Do we need new roles documents?  We need a 
    initial draft in the next couple of weeks and haven't gotten much input 
    yet.
    
    James Kempf comment: People are assuming that there is one solution, and 
    that the entire IETF will adopt it. We should try a bunch of different 
    approaches and see what ones work. This means that we have to be willing to 
    accept that some will not work.  Margaret pointed out that that is one of 
    the listed solutions.
    
    Harald: Fixing our short-term problems is harder than fundamental 
    changes to IETF and may be harmful to the Internet.  If we focus on 
    short-term problems it takes the heat off for five years, but if we don't 
    fix short-term solutions, we're also dead.
    
    Dave Crocker: and what will be the experimental group?  Everything we've 
    done for 13 years has been an experiment revolutionary changes have big and 
    unexpected impacts
    	
    Bob Hinden: Harald is right - we can't fix our problems by 
    fine-tuning.
    
    Harald: if we get to the point where we can't declare consensus, I'll go 
    home.
    
    
    Problem Statement draft - Elwyn Davies
    
    This is a very provisional version (00 draft).  The problems aren't new and 
    aren't (all) IETF-specific, and many are the consequence of growth.
    
    The draft was distilled from 750 pieces of email plus 3 drafts.  The 
    topics were extracted and filed, and in the case of duplicates they were 
    attributed to the person who mentioned them first.
    
    The next steps are to create a visible issue list and publish a second 
    draft by late April, WG last call and IESG review by May.
    	
    Aaron: should be commended for your altruism and suspected for your 
    judgement in taking this role!  However, the draft really bothered me - 
    perceptions start to sound like facts.
    
    [Keith Moore objected to the presentation of material that's already 
    available in the document and that was presented at the plenary, and asked 
    that we focus on discussion rather than presentation.]
    
    Dave Crocker responded to Aaron that it doesn't matter that 
    perceptions sound like facts - we can use this as is.
    
    Graham Travers pointed out that people holding perceptions is a fact, 
    whether the perception is right or wrong.
    
    Chris Allen: there's a cultural element in section 2.1, and mission and 
    vision go hand in hand.
    
    Bob Hinden said that two things were mixed in section 2.2 - WG process and 
    protocol design engineering.  Henning raised the issue of a lack of 
    effective contract with document editors, to which Elwyn added that this 
    leads to lack of delegation.
    
    Margaret: effective engineering processes isn't the goal, it's a 
    solution, and we need to focus on identifying the problem.
    Eric Hickman: over time we apply ourselves less and less, especially 
    towards achieving consensus in adversarial working groups.
    
    Keith: lots of 2.2 emphasis on tools and process, and problem is really 
    more fundamental - groups that can't explain their problem clearly.  Don't 
    compare 2.3 to "good old days" - we're just not sufficiently engaged.
    Christian: can't stay engaged in SIP at 400 msg/day.
    Randy Bush: more people are engaged, but there's more people who aren't
    
    Nathan: Money has gone out of this industry, and more money to be made 
    implementing than inventing
    Steve Trowbridge: Spectators aren't the problem, people frightened away is 
    the problem
    Eric Guttman: people leave mailing lists because they don't want to be 
    insulted
    James: physical violence threatened on one mailing list
    Steve: ADs/WG chairs wait too long to wade in
    Spencer: working group code of conduct?
    Keith: mailing list and meeting methods are very different - should the 
    document split them out?
    	
    Keith totally disagreed with 2.4 as it was written.  It is clear that 
    there is a perception of this, it is clear that a perceptual 
    differences are part of the problem, but he doesn't think it is fact.
    
    Steve Trowbridge said that there is nothing in our procedures that causes 
    this to be the case, it exists in practice.  There are backroom deals and 
    initial windows on work available only to insiders.
    Henning: in the good old days, ADs were a LOT more invisible, now 
    anything of any significance needs AD involvement. 
    
    Margaret said that this section is where the scaling issues of the IETF are 
    showing up.  
    Jonne Soinenen: no single person is the problem, but the process has 
    problems and the perception of problems is equally important as the 
    problems themselves.
    
    Keith commented that no one will disagree with perception that IESG 
    hasn't scaled, but qualifying it as 'authority' is inappropriate. He was in 
    IESG 4 years and they really can't block things. This isn't an accurate 
    statement. He does agree that it's possible for an AD to be 
    capricious and we should fix that.  Marshall answered that there are some 
    former ADs that say that this has happened, in spite of what Keith said.
    
    Eric Guttman: if openness is our goal, why is the IESG is deciding WG 
    formation, document advance, etc?
    Eve Schooler: the IETF isn't more broken than it was ten years ago, 
    that's good news.  People do want to help, and we need to be grooming more 
    potential leaders/insiders.
    
    Margaret said that there are other problems besides scaling and asked to 
    what extent do we want rely on WG consensus vs AD agreement.  Are our 
    selection processes open and fair?
    Jonne: we should see documented discussion of decisions so we can see why 
    they are made.  People just don't know what's going on.  We shouldn't 
    require people to be on the IESG to understand the IESG.
    
    Chris Allen: in groupware field ten years ago, social scientists were 
    looking at mailing lists.  we don't teach people what's already 
    well-known and undersood about our work media.
    
    Pete Resnick: the perception that there are upper class/lower class 
    differences is real.  Distrust is real and can increase if we're not 
    careful.
    
    Margaret: WG chair role isn't well-defined and consistently 
    understood in the IETF.  We need to do role definition before training can be 
    effective.  Chris Allen added that the mentoring process isn't 
    mentioned and is broken.  We have document editors and WG chairs who are 
    brand new.  We should have "shadow" roles who works under old person for 6 
    months/one year for training.
    
    Bob Hinden: There are problems that come with getting new people and 
    readying them to be come the new leaders. I have the feeling that it has 
    become more exclusive. There is a perception of cycling from IAB to IESG, 
    IESG to WG, etc.
    
    Joel Halperin: ADs CAN block things, and sometimes the things we're 
    complaining about are the reasons why we succeed.  Don't create a new 
    problem while solving another one.
    
    Bob Hinden: "blocking" needs to happen at the beginning of the process, not 
    at the end.
    	
    Ted Hardie: Part of the problem is the metaphor "Send up to the IESG" 
    which colors the perceptions. There are many things in the document that 
    have a series of metaphors of consensus, and we use the words of 
    consensus, but in fact, sometimes we use disputation.  When we use 
    disputation with consensus it can cause acrimony.  We need metaphors to 
    match our recommendations.
    
    Dave Crocker said that we may not have structural problems, but rather 
    management problems.  For instance we sometimes choose between two 
    alternatives when we don't have to, and sometimes we don't choose when we 
    should.
    Christian: "Consensus" wasn't in original IETF, but was added in 1992 as 
    response to top-down decision making.  The way we used to solve 
    problems was by building something.  Now we need agreement before anyone 
    builds anything.
    
    Eve Schooler: We can say "here's a good Internet Draft", but "what makes a 
    good WG chair or AD" isn't written down anywhere.
    
    Ted: we've narrowed the point of WGs until it's hard to see the whole 
    picture, then we're surprised when we get problematic point 
    solutions.
    
    
    Wrapup - chairs
    
    We're serious about hitting our milestones.  The problem description is to be 
    in WG last call in May, and the process options document is to be in WG 
    last call in August.  The latter document should be available within the 
    next few weeks.
    
    

    Slides

    WG Participation - The 'Stuckee' Problem
    IETF Problem Statement - Discussion of Draft
    Helping the Internet to Age Gracefully