Re: comments on draft-bierman-sming-ds-02.txt

[Andy Bierman <abierman@cisco.com>]

At 05:25 AM 3/19/2002, Juergen Schoenwaelder wrote:

>>>>>> Andy Bierman writes:
>
>Andy> thanks for reviewing 31 of the 83 pages. That's a good start, I
>Andy> guess.  I will try to respond to the technical questions
>Andy> raised. I think terminology can be finalized after the technical
>Andy> details are resolved.
>
>We have actually read all pages. However, we prefer to stick to the
>fundamental technical contributions at this point in time rather than
>discussing the syntax specification or your view on how SMI-DS
>achieves the objectives stated in RFC 3216.
>
>I also believe that trying to converge on a common terminology makes
>discussions simpler and more efficient. It took some time for us to
>figure out what your terms mean to us (and the same might be true the
>other way round). And I guess this is even more true for people who
>just want to follow the WG process.

terminology and syntax are 2nd order problems at this point.
That doesn't mean it's not important, but since very few people
are putting any effort into this WG, it seems like this can wait
a little bit.


>>> 6. We are not sure whether the goal of 100% forward and backward
>>> compatibility has been reached or to what extend. The SMI-DS
>>> translation to SMIv2 is not really defined in Appendix A and thus
>>> it is hard to judge how much is in fact translatable (and what the
>>> compatibility implication on the wire will be).
>
>Andy> I agree this appendix isn't finished and should have more
>Andy> detail.  Keep in mind that the SMI is a description of object
>Andy> semantics.  It does not specify on-the-wire encoding. 
>
>I agree - on the wire encoding is the wrong phrase. What we meant is
>the assumptions that existing applications and tools have how OIDs are
>stuctured.

I don't agree. The OID semantics are localized. Only the code
for the FOO-MIB needs to know how the OIDs for the FOO-MIB are
structured. Since there are zero MIBs at this time that use
hierarchical naming, there is no impacted code.  If and when
a MIB exists with nest level > 1, the applications and agent code
for that MIB will need to know its structure. Not 1 line of code
for MIBs with nest level <= 1 will be impacted whatsoever.

I agree with you that ConformanceINfo can be just more MIB objects.
I agree with you that the changes I made to the INDEX clause are
not that important, and the old INDEX clause will work just fine.
I do not agree about OID naming. It's not just the manual mapping.
Jeff Case has some great ideas for protocol enhancements to
move aggregates and sub-aggregates. These powerful new ProtoOps
will not work if flat SMIv2 naming is used.  I don't think anybody
will be able to design powerful new ProtoOps if we use flat naming.


>Andy> SMIv2 semantics can be transparently converted to SMI-DS and
>Andy> then transparently converted back to SMIv2, with no on-the-wire
>Andy> changes in either direction.  Just as with SMIv2 -> SMIv1, there
>Andy> are new features for which the conversion will not be
>Andy> transparent.  Converting UNIONs and STRUCTs to SMIv2 is simply a
>Andy> matter of adding lots of verbose text to every affected
>Andy> DESCRIPTION clause, where the phrase "associated instance" is
>Andy> used quite frequently.
>
>You do not say what will be mapped and what will be lost. Lets try an
>example: How does a struct which contains another struct, a union and
>an array look like once translated to SMIv2?

The naming will not be the same for such translations.
Flattening out structs and unions is trivial -- the nodes
are just flattened out into a table (or set of scalars)
and text is added to the DESCRIPTION clauses that defines
the relationship between the nodes (like we do today.
An array becomes a table, perhaps an associated table
if the outside containere was itself an array.

Remember that the official SMIv2 to SMIv1 translation for a
Counter64 is to leave it out completely.  In practice, we
define 2 Counter32s instead.  There is enough room in the
OID namespace to allow for complex data structures and
associated SMIv2 tables to exist for a given MIB. This
is not a hard problem.

The $64 question is whether we want to advance the state of the art
and move forward, or design for the same inefficient, cumbersome
technology we already have. I think the WG has already expressed a 
preference for advancing the SMI technology.  I personally think
it would be a total waste of time to redo the entire SMI if it
doesn't translate to real benefits in the SW that uses the SMI.


>>> 7. It is unclear whether compatibility on the wire between engines
>>> that use SMIv2 style definitions and engines that use SMIng style
>>> definitions (and naming) is still a design objective.
>
>Andy> Please explain what on-the-wire problems exist
>
>See above. We meant compatibility of how existing applications and
>tools expect OIDs to be formed.
>
>>> 8. The proposed new naming touches both the language and the
>>> protocol, at least in the sense that no existing implementation we
>>> are aware of will be able to handle hierarchically named variables
>>> in a meaningful way. We would expect that 100% forward/backward
>>> compatibility implies that I can use existing tools to look at data
>>> implemented by an agent whose implementation is based on an SMI-DS
>>> MIB module.
>
>Andy> The new hierarchical naming is identical to SMIv2 for all the
>Andy> constructs currently available in SMIv2.  For generic
>Andy> applications which have no knowledge of the objects they are
>Andy> manipulating (e.g. mibwalk), all the protoOps work exactly as
>Andy> before, and they still work. For applications that need to
>Andy> understand the objects they use, the semantics of the object and
>Andy> the structure of the instance portion of the OID need to be
>Andy> known.  This is no different than the current situation with
>Andy> SMIv2.
>
>Does this mean a struct which contains another struct, a union and an
>array will just not exist in the SMIv2 mapping? I do not think you
>mean this since it conflicts with the stated requirement in SMI-DS:
>
>   - Maintain 100% forward and backward translation compatibility with
>     SMIv2. [...]

This means that any SMIv2 construct can be translated to SMIv3 and
back to SMIv2 in a transparent way.  Just as with SMIv2 -> SMIv1,
SMIv3 -> SMIv2 will have some features that can be translated, but
not transparently.


>>> 9. We believe that a syntax which looks close to SMIv2 but means
>>> sometimes something different is worse than a syntax which looks
>>> different to SMIv2 and means something different. People who do not
>>> understand the differences will merge SMIv2/SMI-DS in arbitrary
>>> ways. This has happened with SMIv1/SMIv2.
>
>Andy> Please provide examples where this is the case in SMI-DS.  It
>Andy> was certainly my intent to maintain semantics and syntax as much
>Andy> as possible.  I think it would be more confusing to MIB readers
>Andy> and writers to transition to SMIng syntax, in which absolutely
>Andy> every macro and clause in changed.
>
>Frank provided examples. SMIng as well as SMI-DS are trying to make a
>big step forward on how we describe management data. Trying to give
>the impression that this did not happen has more potential to mislead
>users about what they are doing. You obviously noticed while working
>on SMI-DS some some syntactic constructs show be harmonized and so you
>changed some SMIv2 constructs which I am sure others will regard as
>unnecessary change. I guess we can debate this endlessly.

There have been many people that characterize the SMIng syntax changes as
purely gratuitous changes. If we decide to completely change the 
syntax of every construct, then I think we should use XML Schema,
not a new syntax. I don't care about the syntax nearly as much
as the ability to reuse semantics and move aggregates and sub-aggregates
on the wire.


>>> 13. We believe that an array should be similar to an ASN.1 SEQUENCE
>>> OF, that is the array feature should work with an arbitrary
>>> type. The SMI-DS ARRAY mixes containment and multi-valuedness in
>>> one construct.  Programming languages such as C and even ASN.1 do
>>> not do this.  Arrays do not yet exist in SMIng (only in the form of
>>> tables in the protocol mappings).
>
>Andy> Programming languages such as C allow both types of constructs.
>
>No.

typedef struct {
   int   foo[12];
   char  name[30];
} example_t;

struct {
   int   foo[12];
   char  name[30];
} example;


>>> 14. We expect that not only base types, but also derived types
>>> should be allowed in index clauses (just constructed types are not
>>> allowed).
>
>Andy> This is rather problematic and a big change from SMIv2.  How do
>Andy> you INDEX an ARRAY with a UNION, or a complex STRUCT with nested
>Andy> STRUCTs and UNIONs? This capability is not that important,
>Andy> otherwise it would have shown up in the requirements document.
>
>See, we do not use the same terminology and thus we do not understand
>each other...
>
>>> 17. The relationships to SMIv2 are interesting. You change
>>> NOTIFICATION-TYPE to NOTIFICATION for language consistency reasons,
>>> which we have sympathy for. I am sure someone else will see this as
>>> a gratitious change. You also claim that lots of other constructs
>>> do not change as well, including MODULE-COMPLIANCE.  Even if this
>>> is true for the syntax (which I am not sure of), I am sure that
>>> semantics will have to change.
>
>Andy> This is not a gratuitous change.  The syntax structure of SMIv2
>Andy> is "<descriptor> <construct-type> <construct>".  The syntax
>Andy> structure of SMI-DS is "<construct-type> <descriptor>
>Andy> <construct>" which is much more consistent with programing
>Andy> languages.
>
>It is nice to see how it varies over time what you consider a
>gratuitous change. The syntax of SMIng has gone through a process
>of intense discussions and we believe that there are almost no
>gratuitous changes. It is nice to see that you value a consistent
>structure and even something which "is much more consistent with
>programing languages." (I could have written that myself. ;-)
>
>>> 20. It is unclear how much savings the new naming scheme provides
>>> on agents. With SMI-DS, you probably have fewer RowStatus objects
>>> since you do not need them for "contained" multi-valued
>>> attributes. You also can improve code stubs since the compiler now
>>> knows that one table is really just used to hold multi-valued
>>> attributes of some other object. We fail however to understand why
>>> the hierarchical naming simplifies things such as (a) reducing the
>>> degrees of ordering freedom we have in SNMP or (b) reducing the
>>> need to have createAndWait/notInService states in the first place.
>
>Andy> Removing the multiple interacting r/c tables will help reduce
>Andy> agent complexity because the object ancestry is fully specified
>Andy> in every object instance. Using one top level RowStatus (or none
>Andy> at all with new EOS support) simplifies agent code a great deal.
>
>You can do one RowStatus today - but it only works if you can ship all
>the stuff which depends on the RowStatus in a single set request. I do
>not see how SMI-DS naming changes that at all.
>
>Andy> MIB design dictates whether createAndWait or notInService are
>Andy> needed. SMI-DS does change that.  
>
>How? I did not find anything specific on this issue in SMI-DS. Where
>do I have to look at? BTW, the DiffServ MIB also does not require
>createAndWait or notInService and is written in SMIv2.
>
>Andy> It is important to maintain the hierarchical structure of
>Andy> complex data types so generic functions can be written to
>Andy> process them, no matter where they occur for a given usage. 
>
>We agree on this. And SMIng supports that since the generic functions
>deals with the data type - not necessarily with the name.
>
>Andy> The SMIng proposal will require that the SNMP engine have
>Andy> knowledge of every usage of every such construct, and rearrange
>Andy> the OIDs manually to achieve what SMI-DS provides automatically.
>
>I am not sure I agree or disagree. It probably depends on which agent
>toolkit you are used to. You only do it manually if you do not use
>compilers to do all the low-level OID smashing - which I prefer.
>
>Andy> Forcing humans to manipulate the OID ordering in protocol
>Andy> mapping clauses is not workable.
>
>As you said before, there are a few MIB authors but much more MIB
>readers. Explicit mapping to SMIv2 optimizes for the readers - they
>can easily lookup what they get. Using an algorithmic mapping
>optimizes for the authors for the expense of the users (that now have
>to apply the algorithm somehow).
>
>Andy> It means that every time a new object is added to an embedded
>Andy> construct, every MIB that uses that construct has to be updated
>Andy> to assign the new embedded node to a different OID.  This is not
>Andy> consistent with the use of TEXTUAL-CONVENTIONs today.
>
>You can't update textual conventions in this way so I fail to see your
>point. But despite this, I think that manual propagation of changes is
>a feature. I doubt that a system where you can affect hundreds of MIBs
>just by adding something to a reused struct is desirable in practice.
>In analogy to C, if you change a struct, you better recompile all
>modules that use this struct or you are lost since variables now have
>a different memory layout. Our "variables" are SMIng protocol mappings
>and it is thus not surprising that you have to update them. The good
>news is that old implementations will continue to interoperate with
>new implementations - they will just not see the additions.
>
>>> 21. We agree with the idea to hold compliance information in the
>>> agent.  However, we do not understand the value in introducing
>>> "pseudo OIDs". Why are conformance definitions not being kept in a
>>> properly indexed conformance table? Conformance data is just data
>>> and there should be no need for a special mechanism for this
>>> purpose. The pseudo OIDs may give funny results when you do getnext
>>> walks. (Also, the restriction that you can report conformance to
>>> only a single compliance definition might hurt if conformance
>>> statements are revised and you need to report multiple compliance
>>> statement OIDs for continued interoperation.)
>
>Andy> It would be very burdensome to maintain a MIB for the
>Andy> conformance information, for every MIB object that ever gets
>Andy> defined.  Since the conformance data is always the same for
>Andy> every data object, it is easier to align the naming.  I suppose
>Andy> it could be done either way.  A given implementation of a given
>Andy> object instance can only comply with one MODULE-COMPLIANCE, so
>Andy> there is no need to allow for multiple compliance statement
>Andy> OIDs.
>
>We were not talking about a MIB for every MIB object. We were talking
>about a single conformance MIB which works for all object types ever
>defined. Take your SMI-DS fragment and turn it into SMIv2 tables and
>index that tables with the OIDs of object-type macros.
>
>See Frank's response wrt. multiple compliance statements.
>
>>> - We do not understand why there is a need for a new SIZE clause.
>
>Andy> This is a machine-readable indication of the number of instances
>Andy> that are expected/allowed for that ARRAY INDEX.  
>
>Expected or allowed? This is not the same. The examples usually just
>repeat the range restriction - which is kind of useless if the meaning
>is "allowed" and somewhat impractical if the meaning is "expected" and
>the value is 2^31-1.
>
>>> 25. Data structure augmentations:
>>> 
>>> - What is the expected value of introducing SPARSE-CONDITION and
>>> SPARSE-EXPRESSION?
>
>Andy> To provide human and machine readable definitions of the
>Andy> existence relationship between the augmented and augmenting
>Andy> table.  SPARSE-AUGMENTS is a concept that has been around for
>Andy> years, and buried in DESCRIPTION clauses.
>
>Is free form text in a SPARSE-CONDITION clause any different from free
>form text in a DESCRIPTION clause?
>
>>> - SPARSE-CONDITIONS can also go into normal description clauses
>>> since it is just free form text. How do we decide that some text is
>>> so important that it justifies its own clause?
>
>Andy> This existence relationship between tables is used so often that
>Andy> it is worth formalizing.
>
>SMIng formalizes the relationship by way of "extends" and "expands"
>(proposals for better keywords are welcome). SPARSE-CONDITIONS just
>add add another informal documentation clause (I use formal here in
>the sense that machines can read and understand it).
>
>>> - Regarding SPARSE-EXPRESSION, what is the intended usage?  Is it
>>> just documentation clarity? Is the intention to support automatic
>>> test tools? Do these machine readable expressions significantly
>>> simplify/improve code generators? And what after all does the
>>> expression language look like?
>
>Andy> This is intended to be a machine readable version of the
>Andy> SPARSE-CONDITION. The syntax is TBD, but I would like to use a
>Andy> subset of the PolicyScript syntax, found in the PM MIB.  If a
>Andy> simple expression is possible, then tools could take advantage
>Andy> of it. If a complex expression is needed, then this clause would
>Andy> be omitted, rather than making the expression syntax too
>Andy> complicated.
>
>Not sure what a simple expression is. Not sure what a program would
>be able to do with it. I can't say what the value is or is not.
>
>>> 26. Augments Example:
>>> 
>>> - We guess that the SIZE clauses in the definition of inetAddrType
>>> and inetAddr are just cut & paste errors.
>
>Andy> no. this refers to the number of allowed instances of the INDEX
>Andy> component, not the number of octets in an individual instance.
>
>So the SIZE belongs to the whole array and not to the INDEX components.
>
>>> 27. The VAR STRUCT format is unclear. It sometimes contains other
>>> SCALAR/STRUCT/UNION/ARRAY clauses (see ipXStats example) and
>>> sometimes it just contains flat SYNTAX etc. clauses (see myAddress
>>> example).  From reading the grammar, it looks like both are
>>> allowed? What is the difference between both formats in case I just
>>> have a single STRUCT, except that you safe a few lines in some
>>> cases?
>
>Andy> Data definitions are dictated by the problem being addressed by
>Andy> the MIB author. This is no different than a C programmer
>Andy> defining data types for a program. Sometimes they only contain
>Andy> base types, and sometimes they contain some complex
>Andy> types. SMI-DS provides the capability, and it's up to the MIB
>Andy> designer to define appropriate data types for the problem being
>Andy> addressed.
>
>Are you saying: "There is no difference except that you save a few
>lines in some cases"??
>
>/js
>
>-- 
>Juergen Schoenwaelder    <http://www.informatik.uni-osnabrueck.de/schoenw/>

Andy