Summary from MOME interoperability testing event

[Simon Leinen <simon@limmat.switch.ch>]

Here's a more comprehensive summary of the results of the NETCONF
interoperability tests from the week before IETF.  Contributors
include Radu State and Prof. James W. Hong.  The final version will be
integrated into a MOME deliverable which should eventually show up on
http://www.ist-mome.org/publications/
-- 
Simon.

Netconf Interoperability Testing Summary (July 28-29, 2005, Paris, France)

Participants:

1. NEC Labs, Germany
2. WIPRO, Germany
3. LORIA, France
4. POSTECH, Korea

1) Partners tested their Netconf managers and agents using SSH as
   their transport mechanism with respect to a series of predefined
   test cases specified on the MOME website.  Each partner tested its
   manager /agent solution with every other partner.

2) Implementation languages:
   a. One partner used Python for development (LORIA)
   b. Two partners used C for development (NEC, POSTECH)
   c. One partner used Java for development (WIPRO) - verification is needed.

3) The four major problems were identified during the testing were:

A. SSH subsystem for Netconf.

There exist different types of SSH subsystems. Some implementations
used 1) command subsystem, while others used 2) port forwarding
subsystem or 3) custom subsystem (see
http://wwww.eldos.com/sbb/articles/1945.php for categorization).
Using different subsystems by different implementations has made it
difficult to test each other of their Netconf functionalities.

B. Lack of common data model.

There were two problems with the common data model.  Although a basic
data model (related to the interfaces) was specified for the MOME
event, some implementations did not properly implement it.  This issue
was solved on a case by case basis, where manager applications were
customized to fit to a specific agent.  The specified common data
model was network interface.  When an <edit-config> operation was
performed on this, the network interface in the device being tested
was not accessible any more.  Thus, we could not test the
<edit-config> operation.

C. Message separator.

Messages are separated with a special character string(]]>]]>) in the
Netconf/SSH draft.  Some implementations used it, while others did not
use this.

D. Different authentication types in SSH.

Implementations used different types allowed by SSH
(interactive/non-interactive password, public key, etc.) and some
offline agreement had to be done.  Finally however, these issues were
solved.

E. Pipelined Requests.

One implementation allowed multiple requests from a manager.  How to
process this kind of multiple requests were not clearly specified in
the draft.

4) Summary

A. The interoperability testing event was useful to discover who has
   been implementing the Netconf draft and to discover areas to be
   improved in the draft.

B. The results were not too disappointing.  All 4 manager applications
   succeeded to connect and perform requests on two out of the four
   tested agents.  Two agents could not be connected due to missing
   SSH support and delimiter problem (mentioned earlier).

C. Because of different ways of implementing and spending most of the
   time in adjusting to the other implementation, we could not really
   test the Netconf functions or the test scenarios as specified in
   the test spec.

5) Suggestions for the IETF working group

A. Better SSH descriptions and use cases
B. Common data model proposal
C. Better <edit-config> operation descriptions and use cases

6) Suggestions for the developers

A. Continuous exchange of implementation information
B. Remote testing if possible