ecs deployment mode network deadlock with cycles

[luckyworkama]

It's possible in the ecs deployment mode if there are cross-network cycles for the system to deadlock or perform strangely. This happens when tcp sockets fill up and the dfir graph is unable to write to them, this causes execution of the dfir graph to block, which might prevent it from pulling in data from the same tcp socket. This can cause the other node on the other side of the cycle to block while writing to it's tcp socket. Thereby deadlocking the system.

This can be mitigated by using extremely large tcp socket buffers or by introduced an additional queue between the dfir graph and the tcp sockets which has unbounded growth. Neither of these are good solutions.

[richardlee159]

I hit this too during Paxos failover. I think the root issue is that lossless delivery over cyclic communication with bounded memory is generally impossible. You can't have all of them and have to give up something. Some systems explicitly treat network messages as droppable and bound in-flight traffic, relying on the protocol to retry.

Hydro seems well placed to check it as cycles are statically visible in the cross-location graph. Rough idea: sends on a cycle wouldn't typecheck as reliable, so you'd either accept a lossy stream type or write an assume_-style justification that in-flight volume is bounded. The simulator could then also explore drop schedules the way it explores reorderings.

This should also relate to metastable behaviors. Curious whether this sounds reasonable to folks who know the reliability design better.