Re: soft state (was Re: shim6 and bit errors in data packet headers

[Erik Nordmark <erik.nordmark@sun.com>]

Iljitsch van Beijnum wrote:

> > The path from B->A fails.
> > The shim layer on A observes this because it stops receiving packets.
> > (But TCP on A also starts retransmitting.)
> > The shim layer on B thinks everything is fine because it sees the  
> > packets from A, and it sees the ACKs that B is sending back to A.
>
>
> What would happen here is that A doesn't see any return traffic so it  
> starts sending reachability probes. In this case B knows about the  
> problem very soon because the first probe presumably uses address  pair 
> A1, B1 which still works in the A -> B direction.

Yes, so having the shim track whether packets are sent and received 
isn't useful in this case.

> Yes. But as long as we switch from having a short timeout when there  is 
> traffic to having a long or infinite timout when there is no  traffic 
> this shouldn't be problematic.

But you'd need to tell the peer what timeout you are using, right?
So you'd still be adding packets?


> That's true, but it's hard to avoid unless you want to rely on ULP  
> timeouts or behavior very similar to ULP timeouts.

No it isn't hard to avoid. What we need is more or less exactly what we 
have in NUD in RFC 2462 but applies end to end (and as a result using a 
different timeout behavior etc):
 - A probe using the current address pair which is sent when a ULP 
packet is sent, but only if there is no e2e reachability confirmation in 
the last N seconds.
 - Optional ULP positive advice which suppresses the need to probe 
since it provides an e2e reachability confirmation

That is sufficient to tell when there is a problem with the current 
address pair without any extra packets when the ULP provides positive 
advise, and at most one probe exchange every N seconds when the ULP is 
sending packets but doesn't provide positive advice.

In addition we'd need to trigger exploring alternative locator pairs, so 
we would need to decide when that starts (at the same time as the probe 
for the current address pair, or when it looks like that probe might 
have failed?)

> And the amount of traffic would be fairly low. For instance, when  
> traffic is flowing the timer could be 15 seconds. So when there is no  
> more traffic, after 15 seconds the sender would have to send a  
> keepalive. The keepalive could contain a new timer value, which could  
> be X times the previous value (30, 60, 120 and so on), or immediately  
> go to infinity. When traffic starts to flow again, the timer would  have 
> to be reinitialized, or maybe this can happen automatically.

Take a single UDP request/response interaction as an example, but the 
shim doesn't know that there will be a single request/response, and 
where was was some initial TCP traffic that was sufficient to create the 
shim state up front.

The worst case could easily be
1. Send UDP request A->B
2. Send UDP response B->A
3. 14 seconds later: send "timeout is 30 seconds now" A->B
   send "timeout is 30 seconds now" from B to A
4. 29 seconds later: send "timeout is 60 seconds now" A->B and B->A
5. 59 seconds later ...
etc

This means that the total number of packets (when there are no 
failures!) doesn't have a natural bound related to the number of ULP 
packets; 2 ULP packets can result in a large number of "update timeout 
packets" - above there are 6 such packets are a result of 2 ULP packets. 
The only way you can provide a limit of 100% overhead would be to 
immediately switch to infinite timeouts, but then the timeout doesn't 
seem useful?

Compare this with the positive advise ULP triggered probing, and where 
the ULP (in user space) doesn't provide any advice:
1. Send UDP request A->B
   Since the earlier TCP connection just ended the sender has recent
   reachability confirmation for A1->B1, so no probe is needed
2. Send UDP response B->A
   Same as in #1, no probe needed.

If the reachability from the preceding TCP connection is old, we'd get 
the worst-case behavior of
1. Send UDP request A->B
   Reachability confirmation for A1->B1 is >10 seconds old; send a probe
   A1->B1
2. Send UDP response B->A
   Reachability confirmation for B1->A1 is >10 seconds old; send a probe
   B1->A1
3. B receives probe from A; responds
4. A receives probe from B; responds

With the introduction of the DELAY state in RFC 2461 one can probably 
drop the above worst case 6 packets to a worst case of 4 packets. Thus 
the worst possible overhead is 100% (2 probe packets for 2 data 
packets), but this only happens when
 - the application sends a request/response every >10 seconds
 - there is no ULP advice


> For TCP, your positive advice would probably be cleaner, but for non- 
> TCP ULPs which typically don't have a similarly advanced  retransmission 
> mechanism, and especially for ULPs that don't provide  advice, this 
> mechanism, or the other that I mentioned earlier would  probably be a 
> good catch-all backup.

But such ULPs can also provide positive advise if they'd like. I 
actually thought we had added such a capability in section 11 in RFC 
3542, but I don't see it in there.

In any case, it sounds a bit complex to do one thing for TCP and a 
different thing for UDP. I don't know if that was what you were suggesting.

> On the receiver: 5 minutes. On the sender: min: 5 mins + some RTTs,  
> max: 5 mins + N times exponential backoff + some RTTs.

What would happen in this case:
1. One UDP request/response
2. Silent for 5 minutes or so (so a large timeout is conveyed to the peer)
3. There is a failure, for instance in the A->B path
4. Application on A sends UDP to B, which is lost
5. 4 seconds later the app on A retransmits the UDP packet, which is 
also lost
6. 8 seconds later the app on A retransmits the UDP packet; lost
etc

at which point in time would the shim on A and B see a need to probe?

We'd want this to have some relationship to the fact that the 
application is transmitting packets in 4,5,6, and I don't see how the 
receive side timer can do that.
The shim on the sender can decide at some point that packets have been 
sent, but none have been received, but that wouldn't require the 
receiver to run a timer (and having the sender tell the receiver which 
timer to use). But the receive side timer was suggested to avoid sending 
probes on idle connections, so it seems to be necessary.

Unless we take the NUD approach of ULP packet triggered probes that can 
be suppressed by positive advice from the ULP...

   Erik