Re: hierarchical naming

[Andy Bierman <abierman@cisco.com>]

At 07:41 AM 3/14/2002, Juergen Schoenwaelder wrote:

>>>>>> Andy Bierman writes:
>
>>> (b) All existing SNMP applications and tools (except primitive
>>> snmpwalk tools) are based on the notion of conceptual tables and a
>>> "flat" OID space. They won't work with SMI-DS nested naming. To
>>> make them work, you have to "unfold tables" (as you say in section
>>> 8.3 in your ID).
>
>Andy> I don't think this is a significant factor.  One could claim
>Andy> many years ago that no existing applications understand
>Andy> Counter64 (or even Unsigned32) so these data types cannot be
>Andy> added. Applications have to change to use brand new
>Andy> features. IMO, the advantages of hierarchical naming outweigh
>Andy> the disadvantages.
>
>What do you think are the advantages? How does hierarchical naming in
>the OIDs reduce the costs of implementing agents with current protocol
>operations?

1) Hierarchical naming allows more powerful and comprehensive
   data structures to exist. If the OIDs are flattened out,
   then you can't represent embedded ARRAYs. Simple data structures,
   such as an array of port stats within an array of interface stats,
   the RMON2 usrHistoryControlTable/ObjectTable or an access control 
   list that contains some base types plus an ordered list of access 
   control rules cannot be represented as a single data structure.  
   With SMIng, any data relationship in which not all members have a 
   1:1 existence relationship to each other have to be separated into 
   multiple structures. This leads to more complex instrumentation code.
   Flattening out the attributes only works reasonably well for
   simple structs in structs. 

2) Manual flattening of complex data structures will likely be 
   labor-intensive and error-prone. A MIB writer will have to
   recursively iterate through all the levels of the data
   structure and create sequential OID assignments for them.

3) It is difficult to analyze OIDs by inspection, but this has to be
   done for debugging. A complex data structure which is flattened
   out somewhat arbitrarily will be difficult to reconstruct by hand.
   
4) Agent instrumentation and engine code will be able to dispatch
   by subtree, allowing for more modularized implementation. Using
   the ACL example, if the array of ACL rules if aclList.4, then
   that subroot can be dispatched to a single modular handler that
   processes the rules, even if the rule is made up of multiple
   attributes. The handler for the ACL name (e.g. aclList.2)
   does not need to know anything about the structure or indexing
   of the ACL rules. This is one of the goals of data abstraction.


>>> (c) Since SMI-DS assumes RFC 1905 protocol operations, you can't
>>> get/set "complex" objects. You still ship lists of varbinds of the
>>> leaf attributes - the same list of attributes you would be shipping
>>> with SMIng or SMIv2. Only the OIDs differ (usually making the
>>> messages a bit longer, but that is not relevant here).
>
>Andy> The same is true for SMIv2, SMIng, or SMI-DS. Macros for
>Andy> not-accessible constructs cannot be transported.  The purpose of
>Andy> the SMI is to express management information in the most
>Andy> efficient and interoperable manner possible.  I don't see what
>Andy> ProtoOps have to do with the evaluation of different SMI
>Andy> approaches.
>
>The structural aspects are almost the same in SMIng and SMI-DS. (We
>have some differences in the details but I am sure there is room to
>work them out). The hierarchical naming is of course the major
>difference. From the language point of view, it is easy to adapt SMIng
>to do hierarchical naming as well.
>
>My point is that hierarchical names IMHO give real benefits if I can
>use them to access (nested) objects rather then simple variables that
>represent aspects of a "virtual" object. It makes a difference whether
>I can create a complex object (preferably in one and only one way) or
>whether I have a bunch of variables that I can ship in arbitrary
>orders spread across multiple transactions in order to create the
>complex nested object. Without protocol operations that take advantage
>of the fact that hierarchical naming provides names for "objects"
>rather than just "variables", I just fail to see the big win.

I think the list above show real benefits, even for the crude
capabilities of SNMP.  If and when EOS or some other protocol
allows for aggregate and sub-aggregate transfers, the benefits
will be even more apparent. 


>Note: I am not at all negative about your work and nested naming. I am
>just looking to understand where the gain is when we use hierarchical
>names with our existing SNMP.
>
>>> (d) So the question boils down to: What do you gain by having
>>> hierarchical names in the OIDs (under the assumption of RFC 1905
>>> protocol operations)? I did not yet find a good convincing answer
>>> to this question. Note that a code generator which understands
>>> SMIng does know the data organization of the nested structures
>>> before they were flattened out and thus it can generate code to
>>> reconstruct the nested data structure from the varbind list without
>>> requiring the nesting information to be present in the OID. I did
>>> not find a way to do the same thing dynamically at run-time since
>>> you generally do not know where the OID nesting ends and the
>>> instance identifier starts.
>
>Andy> I don't think it's reasonable to assume the SNMP engine/toolkit
>Andy> will efficiently rearrange nested sub-aggregates, to allow for
>Andy> the same type of reusability that could be achieved if instances
>Andy> of the same complex data type were named in a consistent
>Andy> (hierarchical) manner. The engine may not know where the
>Andy> instance OIDCs start, but the reusable code to validate a
>Andy> specific data type does.
>
>My assumption is that a reasonable SNMP toolkit converts received
>varbind lists into implementation language data structures derived
>from MIB modules before any callbacks are invoked. Right now, all such
>a tool can generate (without additional hints) are simple structures.
>With SMIng and SMI-DS, we can define in the MIB module more complex
>data types and thus a smart toolkit will take advantage of this and
>generate code to convert received varbind lists to more complex
>implementation language data structures. And since SMIng and SMI-DS
>allow to reuse (complex) data types, chances are good that smart
>toolkits can produce more compact implementation code. With current
>SNMP operations, all this happens at compile-time and at run-time, all
>you need are unique OIDs - I think it does not really matter much
>whether whether they contain some information about the nesting or
>not.

I am concerned about the complexity of this engine code, and
even more concerned about the increase in codesize that will
occur if lots of 'OID rearrangement rules' are stored on
the agent.


>Perhaps I am just missing something. Do you have an algorithm in mind
>which you can outline in pseudo-code which shows me how a command
>responder (which current SNMP protocol operations) can take advantage
>of having hierarchical OIDs when it processes a varbind list?

not yet


>>> (e) The SMI-DS ID says in section 5.5 that hierarchical naming can
>>> help to reduce implementation complexity for agent and application
>>> developers for SNMP Set operations. The only benefit I can see is
>>> that you can reduce a few RowStatus objects by the expense of
>>> having larger set operations. I do not see how the hierarchical
>>> naming scheme affects the complexity of having to support
>>> arbitrariluy ordered and arbitrary complex set operations and the
>>> issues with createAndWait and notInService states.
>
>Andy> I think the ease-of-use factor is important. 
>
>Ease of use for whom? The MIB author / reader?

writers, readers, and developers


>Andy> The current SMIng drafts require manual and explicit unfolding,
>Andy> flattening, and mapping of the natural data structure into an
>Andy> SMIv2-style row.
>
>Yes.
>
>Andy> I don't see how SW will efficiently take advantage of the
>Andy> hierarchical nature of the data structures once this is
>Andy> done.
>
>As I said before: Programs that generate manager or agent stubs now
>know what the complex structures were that the MIB author had in mind
>and can do something useful with this information. And old programs
>that do not understand SMIng will continue to function since the OID
>assignement basically defines the conversion to SMIv2. Furthermore, it
>is possible to map existing SMIv2 MIBs into SMIng and at the same time
>providing more information so that you can take advantage of this.
>Frank and I have been very conversative because we believe that
>forward and backward transitions are very important give the existing
>number of SMIv2 MIBs the community has defined in the last several
>years.
>
>Andy> This important relationship should be self-describing
>Andy> (like XML). I think new applications that wish to take advantage
>Andy> of new SMI features will be more hindered than helped by
>Andy> flattening complex data objects.  Both SMIng and SMI-DS agent
>Andy> code will easily co-exist with SMIv2 agent code, because the
>Andy> <oidBase> structure is identical, and SNMP engines dispatch by a
>Andy> longest <oidBase> match of the registered instrumentation
>Andy> handlers.
>
>With the current SNMP and all the many applications and toolkits that
>assume a flat OID naming system based on rectangular tables, your
>agent will have to provide flat OIDs in addition to hierarchical OIDs
>for a long long time. Without protocol operations that take advantage
>of the fact that we now can address complex objects, I just fail to
>see the savings.
>
>>> With my current understanding (which might miss important pieces -
>>> if so, please correct me), weighting (b) against (d) and (e), I
>>> think that the current SMIng approach provides a much better
>>> transition path since we can continue to use all existing tools but
>>> we can improve them to be much smarter in efficiently handling
>>> complex (reused) data structures.
>>> 
>>> Sure, if you break (a) by defining new protocol operations which do
>>> not anymore operate on a list of lexicographic variables but on
>>> structured objects, things will be much different. But we do not
>>> have such a protocol nor do we have proposals underway in EOS nor
>>> does the charter tell me that we should work under the assumption
>>> that we have such a protocol.
>
>Andy> I guess it's a judgement call as to which type of new code is
>Andy> easier to write, and which will have a smaller footprint.  We
>Andy> seem to agree on the value of hierarchical data structures, but
>Andy> not how they should be mapped to OIDs. This is significantly
>Andy> further along than we were at the last IETF.
>
>As I said before: I am not totally negative or against hierarchical
>names. I just want to make it clear that I believe this step has
>serious implications on the existing applications/tools that assume a
>flat tabular naming style and that it only plays off its full
>potential if there is support for "complex" objects rather than simple
>typed variables in the protocol.
>
><soap>
>  If the IETF (not this WG) would be going down the path to manipulate
>  objects rather than lists of typed variables, then you can ask
>  whether this is still SNMP. Sure, today's Ethernet works quite 
>  different from the good all yellow cable and it is still called
>  Ethernet. But Ethernet had a good branding while SNMP ...]
></soap>
>
><not-so-soap>
>  The cool thing about SMIng is that it allows to map to protocols
>  which manipulate objects rather than lists of typed variables by
>  doing another protocol mapping. ;-)
></not-so-soap>
>
><much-more-soap>
>  Perhaps it is time to redo COPS-PR by creating COPS-NM - a protocol 
>  which ships real nested objects instead of just flat table entries.
></much-more-soap>
>
>/js

Andy