Retry failed snapshot submission

[seanlinsley]

Snapshots are now submitted to a Queue that retains recent snapshots if submission fails. This significantly reduces the risk of data loss when there are temporary network connectivity or API availability issues.

[seanlinsley]

Is LocalDir still in use?

[seanlinsley]

I dropped this function primarily so that the snapshot UUID and collected timestamp no longer need to be retained. /v2/snapshots/compact is no longer used, but what about /v2/snapshots/test?

[lfittl]

As discussed, an open question here is whether we should store this in temp files or not - if we can reliably assume that full snapshots are only a handful of MBs in size (which I think is unlikely) it might be fine to stay in memory, otherwise we should utilize temp files (at least when the snapshot is big) to avoid high peak memory use when the server side is temporarily unavailable.

We could also consider having a hard limit of how much memory is used across all servers on the collector for this purpose to avoid OOMs, if there is a good reason to not use temp files.

[seanlinsley]

This PR was essentially rewritten. Notable changes:

• All servers now respect a global 200 MB max queue size.
• A single queue per server is used to avoid significant complexity needed to coordinate two separate queues (previously there were separate queues for compact snapshots and full snapshots). As a consequence the max per-server queue size was increased from 10 to 200 to ensure we retain at least one full snapshot for a reasonable amount of time.
• Snapshot submission over the websocket now returns an error, so we're able to correctly retain snapshots in the queue.
• There are now tests to validate the queue's behavior.