h2: `RST_STREAM` for a stream rejected in `FILTER_PROTOCOL_CONNECTION` sits unflushed until other traffic forces a write

[johanneshauer]

When a user callback rejects a WebSocket handshake by returning non-zero from
LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION, HTTP/1.1 closes the connection immediately,
but an RFC 8441 extended-CONNECT stream gets nothing at all: no HEADERS, no
RST_STREAM, no GOAWAY. The client sees silence until the idle timeout takes the whole
connection down.

The reset is not missing — it is generated and then not written. Any unrelated frame
that forces a write on the connection flushes it out at that instant, seconds late.

Environment: libwebsockets 4.3.5, as statically linked into Debian/Ubuntu's
ttyd 1.7.7 package — not the libwebsockets19t64 shared library. ldd /usr/bin/ttyd
shows no libwebsockets.so at all; both report 4.3.5-unknown, so it is the same
source, but swapping the shared library will not change what ttyd exercises. Built with
NET CLI SRV H1 H2 WS. ttyd's FILTER_PROTOCOL_CONNECTION callback returns 1 for a
cross-origin handshake; any server rejecting there should behave the same.

Because the lws is baked into that binary, this reproduction is pinned to 4.3.5 — I
cannot retest against v5.0.0 without rebuilding ttyd against it, which I have not done.
If this is already fixed on main, that would explain it and I am happy to close.

Reproduce

Server: any lws server whose FILTER_PROTOCOL_CONNECTION returns non-zero. I used
ttyd --ssl --ssl-cert c.pem --ssl-key k.pem --check-origin --credential t:t /bin/cat
on loopback, and sent an extended CONNECT with a foreign Origin so the callback
refuses it.

A — refuse, then stay passive. The client sends the CONNECT and then does nothing:

  0.000s  ALPN=h2
  0.000s  >>> stream 1: extended CONNECT   (callback refuses it)
  5.003s  [tcp] server closed the connection (FIN)

Nothing is received for the whole 5 s. No RST_STREAM, no GOAWAY.

B — identical, except the client sends an unrelated GET at t+3 s:

  0.000s  ALPN=h2
  0.000s  >>> stream 1: extended CONNECT   (callback refuses it)
  3.004s  --- nothing so far. now sending an unrelated GET on stream 3 ---
  3.004s  <<< RST_STREAM    stream=1  len=4  err=0
  3.004s  <<< HEADERS       stream=3  len=126
  3.004s  <<< DATA          stream=3  len=17
  8.010s  [tcp] server closed the connection (FIN)

The RST_STREAM for stream 1 arrives in the same millisecond as the response to
stream 3 — three seconds after the rejection that produced it. That is what makes this
look like a queued-but-unflushed write rather than a missing one.

HTTP/1.1 control, same server, same callback, same foreign Origin, ALPN forced to
http/1.1:

  0.000s  connection closed by server

Immediate. The server-side log records exactly one rejection in each case, so the
callback is firing identically on both transports.

Effect

Any rejection on an h2 connection — bad credentials, a disallowed path, a failed origin
check — presents to a browser as a multi-second stall rather than a refusal. In the case
that led me here, that turned a deterministic, correctly-refused handshake into
something that looked like a flaky network for a long time. Chrome recovers only because
the connection eventually dies and it retries without an h2 session to reuse.

Also, possibly deliberate

The reset carries err=0 (NO_ERROR). RFC 9113 §8.8 has NO_ERROR in RST_STREAM
meaning the stream was closed without error, which is how a cancellation is signalled;
a rejection a client may safely retry elsewhere is what REFUSED_STREAM is for, and it
is the code that tells a client the request was never processed. If the intent is "this
stream is refused", REFUSED_STREAM would carry that; I may be misreading the intent,
so treating this as a question rather than a report.

Notes

I reproduced this through a third-party server rather than a minimal lws example,
because that is where I hit it. If a self-contained C repro against
minimal-http-server-* would be more useful, I am happy to write one — likewise a
capture with -d 15 if the internal path is not obvious from the above.

[johanneshauer]

Edited the environment paragraph above for accuracy: the lws under test is statically linked into Debian's ttyd, not the packaged shared library, which is also why the repro is pinned to 4.3.5. Offer of a minimal minimal-http-server-* repro against current lws stands.