This is one of the points I've been hammering on extensively a while back: you can't trust that the source address belongs to the actual correspondent, and you also can't trust that it's reachable. So both for security and robustness, a host has to determine which destination address it's going to use to reach a correspondent regardless of the original source address.
An option for this is to verify the proposed new address contained in the
source address of incoming packets before using it as a destination address
for outgoing packets.
In any case, this may mean that the receiving host may
not be using the received source address as destination address for outgoing
For establishing a communication, when the first
packet is received, i guess that the option of using the received source
address as destination address of reply packets seems attractive. Otherwise
you would need to discover at least one locator of the other party through
alternative mechanisms, like a directory, based for instance in the
identifier, which would at least add some latency to the process.