This is a good point, and I don't recall seeing it in the draft.
However, I would be interested in hearing if you have ideas what the
protocol should do after narrowing the flow label down to 3 or 4.
How does it disambiguate further from those 3 or 4? Would you rely
on the implementation taking a peek at (say) layer 4 headers or some
other information?
There are two obvious options:
1. The flow label is relevant regardless of the addresses: this can't
work because of the possibility of clashes with flow labels in
unrelated packets
2. The flow label is only relevant with certain source/dest address
combinations: but what's the additional benefit of the flow label
here, we can demultiplex on the addresses