RE: AD review of: draft-ietf-tewg-measure-05.txt

["JACQUENET Christian FTLD/IAP" <christian.jacquenet@francetelecom.com>]

Dear all,

Additional comments below.

-----Message d'origine-----
De : owner-te-wg@ops.ietf.org [mailto:owner-te-wg@ops.ietf.org]De la
part de Wijnen, Bert (Bert)
Envoyé : samedi 26 avril 2003 18:24
À : Ash, Gerald R (Jerry), ALABS; Tewg (E-mail)
Objet : RE: AD review of: draft-ietf-tewg-measure-05.txt


WG,

I'd like to hear/read more comments/feedback on my review.

Inline responses to the comments from Jerry

Thanks,
Bert

> -----Original Message-----
> From: Ash, Gerald R (Jerry), ALABS [mailto:gash@att.com]
> Sent: vrijdag 25 april 2003 23:43
> To: Wijnen, Bert (Bert); Tewg (E-mail)
> Cc: Ash, Gerald R (Jerry), ALABS
> Subject: RE: AD review of: draft-ietf-tewg-measure-05.txt
>
>
> Bert,
>
> Let's not throw the baby out with the bath-water.  The I-D
> serves a critical need and should go forward.
>
Can you formulate the "critical need" that is being served?

CJ: the specification and the enforcement of a traffic engineering policy
(or a set of) rely upon the activation of complex features, hence yielding
the critical need to qualify how well is such a policy enforced and how
efficient it is, from both a network resource optimization standpoint (hence
affecting the network planning policy) and from a QoS standpoint (like the
provisioning of (hard) guarantees (loss, inter-packet delay variation,
one-way transit delay, etc.) as far as the level of quality associated to a
given service is concerned). Qualification of the efficiency of such
policies obviously rely upon measurement techniques and parameters, and
IMHO, this draft depicts a pretty good set of requirements that not only
reflect the aforementioned (and defined) critical need, but which also
provides an accurate reference/cornerstone for the definition of TE
measurement framework and tools. That's the reason why I fully support this
draft.

> Regarding your comment:
> 'I would expect a CRISP set of "requirements for additional
> measurements, configurable/negotiable parameters/controls"
> ... but not the extensive exploration and text that I now see'.
> The requirements *are* there for additional measurements,
> configurable/negotiable parameters/controls, but they need to
> be made more prominent and crisp (as you say).
>
Well cedrtainly I could not see them... they are too well hidden
I guess :-(. So making them CRISP is certainly a good thing.

CJ: I'd like to hear other comments about the readability of the contents of
this draft. The fact you find these requirements hard too find does not
necessarily mean that all the reading community (including IESG members)
shares this point of view.

> I'd suggest to focus the main body of the I-D on these
> requirements/recommendations, and move the descriptive
> material to Annex(es).  The text has evolved over a long time
> with many comments and responses, as such it has gathered
> some moss along the way...  But the critical essence is
> there, and the requirements *need* to move forward to provide
> essential TE measurements.
>
So.... it seems we agree that this title:
    A Framework for Internet Traffic Engineering Measurement
is not good and should be something aka
    Requirements for essential TE measurements, parameters and controls
and then of course the main body of the document should focus on
that and be clear as to waht is a real requirement and why.

CJ: "Requirements for TE measurement" should be sufficient.

> Regarding your comment:
> 'if we were not yet doing any of it... then I wonder if this
> would be helpful at all.'
> There are many important requirements/recommendations in the
> I-D.  As one example, the ability to collect traffic data to
> generate a traffic matrix (e.g., 'node-pair data', see Table
> in Section 10.1) is woefully lacking.  To get a flavor of the
> problem/need, see these papers on the topic (links to papers
> at http://www.research.att.com/~jrex/papers.html#tmeas):
> A. Feldmann, A. Greenberg, C. Lund, N. Reingold, J. Rexford,
> G. True, "Deriving traffic demands for operational IP
> networks: Methodology and experience," IEEE/ACM Transactions
> on Networking, June 2001, pp. 265-279. (An earlier version
> appeared in Proc. ACM SIGCOMM, August/September 2000.)
>
I'd like to see in the document something aka:

   REQUIREMENT: ability to collect traffic data to generate trafix matrices
   DESCRIPTION: describe the details as to what traffic data is needed
   JUSTIFICATION: describe what the problem is that needs to be solved
   PRIORITY: x

Don't need to dwell too much on that... but have a concise and crisp
writeup. Maybe RFC3216 can serve as a good example.

CJ: it's basically a matter of presenting things (and I don't like the
notion of priority you seem to intriduce in your suggestion, because (1) the
so-called importance of a requirement may very well vary from one operator
to another (because of the range of service offerings actually supported,
because of the network topology, etc.), and (2) priority generally implies
classification and reference, which to me, is pointless as far as a
requirements document is concerned - see RFC 3139, for example).

> This work shows the difficulty in deriving a traffic matrix
> with today's measurements.  A traffic matrix is basic to any
> network engineering or planning task.
>
> Other critical traffic/performance measurement requirements
> identified in the I-D are also lacking and need to go forward.
>
> A few more comments below.
>
> Jerry
>
> > - a lot of text... but not very focused and to the point
> (or at least I have trouble seeing the main points)
> > - not a "framework", rather an exploration of TE measurement
> >   related topics (more like  a summary-introduction)
>
> As above, the I-D serves a critical need.  The text can be
> focused on the requirements/recommendations, as suggested,
> and the descriptive stuff moved to an Annex.
>
One could wonder how much of the descriptive stuff is really
helpful or usefull. But if it is in an ANEX, then maybe that is
OK. I would say however, that if we have the requirements
crisply formulated, with description, justification and priority,
then one would hope that a lot of additional text is not needed
(not even in an anex). Think about the audience. It would be
protocol developers and/or vendors who have to understand the
requirement and be able to make modifications to their protocol
and/or products to meet the requirement.

CJ: your suggestion of putting examples in an appendix is fine, but I'm
still not convinced of the efficiency of your proposal (description,
justification, etc. - see above comment), simply because you personally
found this document hard to read.


> > - I am not an operator, but I think if I were one, then
> >   - if we were already doing TE measurement stuff (most
> >     likely) then reading it seems a waste of time
> >   - if we were not yet doing any of it... then I wonder if
> >     this would be helpful at all.
>
> Operators have to get by with what is available, what else
> can we do?  This does not mean operators are already doing
> 'TE measurement stuff' in an adequate way, and that nothing
> further is required.  E.g., see comments above on derivation
> of a traffic matrix.
>
I would hope operators can also understand and support the
requirements. If not, then I really doubt that they are valid
requirements. And they must understand them, so that together
the operators can say to protocol developing WGs and to vendors
that these are the set of requirements to be met.

CJ: I fully understand and support the requirements described in this draft.
And I'm pretty sure I'm not the only operator's representative who fully
understands and supports the contents of this draft.

> > - W.r.t. the TEM WG work item, it says:
> >      The tewg interacts with the common control and measurement plane
> >      working group to abstract and define those parameters,
measurements,
> >      and controls that traffic engineering needs in order to engineer
> >      the network.
> >   So I would expect a CRISP set of "requirements for additional
measurements,
> >   configurable/negotiable parameters/controls" ... but not the extensive
> >   exploration and text that I now see. Why do people (or the WG) think
> >   that this document meets the WG deliverable for TEM ??
>
> The requirements are there, and are critically needed.  They
> need to be brought out in a clear and crisp way, as suggested above.
>
Yea.. you keep repeating that.

CJ: being ironic and cynical doesn't help the discussion that much.

> > - W.r.t. review:
> >   - 4 people from ATT support it. Waisum is one of them and is main
editor.
> >     Others (Nick Duffield, Bob Cole) have some of their material listed
in
> >     the doc.
>
> So AT&T is stuffing the ballot box?

That was not what I am saying. My point was more: it is clear that those
people who have contributed support the document. Would be strange if they
did not, would it not? But I can see now that maybe I should have made
my statement a bit different.

> I'm mentioned in acknowledgements, others in references, are we
> disqualified to comment?  A certain set of people (*individuals*)
> who are  the ones most interested/knowledgeable are the ones also
> most likely to comment.
>
> >   - 3 EDU users commented, 2 said they found it a good doc
> >     3rd one asked a few questions
> >   - Raymond Zhang is positive. He is from info.net ???
> >     is that an operator?
>
> yes.
I thought so but was not sure. Maybe Raymond can answer himself, no?
Raymond, can you fill me in what sort of network you operate?
Feel free to do so privately if you prefer that.

CJ: I'm not sure I understand what you're trying to get to here. Are we
supposed to *justify* our position and/or our networking conditions to have
the right to write and submit a document to the IETF? I'd like to hear  the
opinion of IESG representatives on this subject.

>
> >   - Richard Tibbs gave a thumbs up, he is from oakcitusolutions.com.
> >     What role/function does he play/have? Operator, code/tool-developer?
>
> co-author.
>
Ah... ok, so obvious that he would support it. Should have been in
my first list of people above.
You did not answer (but better if Richard does himself) if he is
and operator or what role plays in his daily life. The reason I ask
this is because I want to udnerstand their background and the eyes
and context that they probably have used to review.

> >   - One hotmail user (Spyrokontigiogis) gave a thumbs up. Not that
> >     he/she added any comment. Do we know him/her?
> >     What role/function does he/she play/have? Operator,
code/tool-developer?
>
> Level-3 employee?
>
Can you be more specific? Is he/she in the accounting dept? I guess not.
But again, I want to understand the context within which the review was
done.

CJ: as far as the document's review is concerned, it has followed the basic
rules - submitted to the tewg WG mailing list, waited for comments, issued
the updated version, gone into a WG last call procedure and then submitted
to the IESG for review. Why are you *demanding* the background, the position
(maybe the skills, resumes while you're in this questioning process?), etc.
of the people who took some of their time to writte and/or review this
document and possibly make some comments/suggestions? What's so specific
with this document that justifies this kind of inquisition?
> >   - Dimitri Papadimitrou (Alcatel) asked a question/suggested some text.
> >     I did not see if he likes the doc or not
>
> did not object.
>
> >   - Blain Christian (uunet, so maybe a real operator?)
> >     withdrew as co-editor
> >     That does not sound good (in my view)
> > - So where are the real operators that support this?
>
> So *individuals* from AT&T, Infonet, France Telecom, and
> Level-3 (?) supported this, IETF does not have
> companies/operators positions.  Also, are you saying that the
> above operators are not 'real' operators versus other
> operators who you think are real operators?
>
Sorry for the bad wording. When I use company names, I try to
see how many differnt viewpoints from people active in different
types of networks have looked at it. From their operational experience.
I do not consider them as speaking for their company, but indeed as
individuals. I'd like to see comments/review/support from all
types of operators, that is carriers, telecom operators, cors ISPs,
Access ISPs, small ISPs, Enterpise Networks... etc.

CJ: why don't you clearly state what you dislike about this document instead
of demanding an echo from all the Internet community (you should also
include end customers, since TE may have an impact on the level of quality
of the service they've subscribed to, hence imposing service providers that
they fulfill their commitment by providing the appropriate measurements on a
regular basis)?

> > My thought is that I can do two things:
> >
> > - send back to WG and say that this is "not good enough" and
> >   that I do not
> >   feel comfortable to present it to IESG for approval.
> >   It does NOT meet (in my eyes) the WG deliverable for TEM.
> > - send it to IESG with my recommendation to NOT approve for
> >   same reasons.
> >
> > So let me try the first option first and ask the WG what they
> > have to say to my review.
>
> I find it rather disappointing that you waited this long to
> come down on this I-D so hard.  If you had this level of
> concern, I think you should have tipped your hand long ago,
> but perhaps this is first time you have read the draft.

That is a valid complaint. We are aware that a lot of the AD/IESG level
review takes place too late in the process. Not easy to fix, we (ADs)
all have trouble to keep up with what we have on IESG agenda, let alone
to dive into many WG docuemnts early. We try to improve... but in this
case I missed it.

Now I do understand that the WG chairs have been pushing back on this
in the beginning quite a bit too. I believe that the doc has improved
quite a bit since the earlier versions... but when I review it in
the context of the WG deliverable, then I am clearly still very
dissappointed. If you say that all the "requirements" are hidden in
that doc, then pls try to make them clear. It seems that needs major
rework.

> We have been admonished many times to read the draft and comment
> on this list, and many folks have abided by your wish.
>
I guess it depends on what you count as "many". When I was a kid,
we joked about people counting:  1,2,3,many. Mmm.. maybe that statement
is not appropriate.

CJ: this is the kind of argument we could also put forward when considering
other initiatives, such as the recent creation of the netconf WG for
example, based on an individual contribution that didn't get too much echo.
In this case, surprisingly enough, I can see that a silent majority was good
enough to launch yet another WG (hence additional contributions).

But when, say, 5 to 10 people have expressed their opinions on a
*requirements* document that is part of the tewg WG charter, and that this
document has been updated accordingly so that yet another silent majority
didn't make any specific objection so far, then the document is judged of
bad quality by a single person.

This is simply *unfair*, especially when this person is ready to consider
specifications of solutions (I'm referring to the inter-domain TE stuff -
see the IETF56 tewg WG minutes) *before* the accompanying/reference
requirements document has been accepted by the tewg WG as a WG document, and
discussed by the WG members.

Cheers,

Christian.

> If we discard this I-D, the TEWG charter item also will go
> unfulfilled.  That's bad. I think the I-D serves an important
> need, and can address your comments, as above.
>
But I'd rather have us confess that we were unable to deliver on a
WG item than us delivering something that is not in good shape
(or bad quality, which I think is the case).

Hope this explains my posting a bit better.
Bert