Skip to content

fix(rabbitmq): bound a publish so it cannot hold a transaction open (#7630) - #7631

Open
Romuald Lemesle (RomuDeuxfois) wants to merge 1 commit into
mainfrom
fix/bound-rabbitmq-publish
Open

fix(rabbitmq): bound a publish so it cannot hold a transaction open (#7630)#7631
Romuald Lemesle (RomuDeuxfois) wants to merge 1 commit into
mainfrom
fix/bound-rabbitmq-publish

Conversation

@RomuDeuxfois

@RomuDeuxfois Romuald Lemesle (RomuDeuxfois) commented Aug 27, 2026

Copy link
Copy Markdown
Member

Proposed changes

Sending an inject to RabbitMQ happens inside its database transaction, and nothing made that send give up. Java has no socket write timeout, so a broker that stops reading parks the thread forever, holding a connection until the pool is empty and the platform stops answering.

  • publish() runs on a separate thread and gives up after openaev.rabbitmq.publish-timeout-ms, 30 s. Only the inject fails.
  • Connection timeout 60 s → 10 s.

Contains the damage, does not remove the cause (#7630).

Checks done

ExecutorTest, HealthCheckServiceTest and SimulationInjectApiTest pass. Nothing tests the timeout itself.

Copilot AI lite review requested due to automatic review settings August 27, 2026 12:13
@github-actions

Copy link
Copy Markdown
Contributor

📖 Documentation check⚠️ Passed with warnings

3 functional file(s) changed, 0 documentation file(s) changed.

Suggestions (non-blocking)

  • 🟡 New configuration propertyopenaev-model/src/main/java/io/openaev/service/RabbitmqService.javadocs/docs/deployment/configuration.md

Add the No need documentation label to bypass this check when no documentation update is needed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds containment mechanisms around RabbitMQ publishing so that a stalled broker can’t indefinitely block basicPublish and keep database transactions (and therefore row locks / the Hikari pool) held open, preventing platform-wide outages during broker disk/memory alarm conditions.

Changes:

  • Add connection-level timeouts/heartbeat on the shared RabbitMQ ConnectionFactory.
  • Run RabbitmqService.publish() on a separate executor and enforce a publish timeout (configurable via openaev.rabbitmq.publish-timeout-ms).
  • Add a focused test suite for publish timeout vs healthy/error broker behaviors.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
openaev-model/src/main/java/io/openaev/service/RabbitmqService.java Adds bounded-time publish via an executor + timeout and shuts the executor down on bean destroy.
openaev-model/src/main/java/io/openaev/driver/RabbitmqDriver.java Adds connection timeout / handshake timeout / heartbeat to fail fast on unresponsive brokers.
openaev-api/src/test/java/io/openaev/service/RabbitmqServicePublishTest.java New unit tests validating timeout behavior, success path, and surfacing broker IO errors.
openaev-api/src/main/resources/application.properties Introduces openaev.rabbitmq.publish-timeout-ms with documentation.
Suppressed comments (2)

openaev-model/src/main/java/io/openaev/service/RabbitmqService.java:85

  • issue (blocking): newFixedThreadPool doesn't actually cap the number of pending publishes (only the number of active threads). Use a ThreadPoolExecutor with a bounded queue (or a SynchronousQueue) so publish requests fail fast once the pool is saturated, keeping memory and downstream load bounded during a broker stall.
  private final ExecutorService publishExecutor =
      Executors.newFixedThreadPool(
          PUBLISH_THREADS,
          runnable -> {
            Thread thread = new Thread(runnable, "rabbitmq-publish");

openaev-model/src/main/java/io/openaev/service/RabbitmqService.java:147

  • issue (blocking): once the publish executor is saturated (all threads stuck + queue full), submit(...) will throw RejectedExecutionException. Today that would bubble as an unchecked exception; it’s better to convert this into a controlled failure (e.g., TimeoutException) so callers consistently roll back the inject instead of failing with an unexpected runtime error.
    Future<Void> pending =
        publishExecutor.submit(
            () -> {
              doPublish(injectType, publishedJson);
              return null;

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +20 to 25
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
Comment on lines +49 to +69
@Test
@DisplayName("given a broker that never answers should give up instead of blocking the caller")
void given_brokerThatNeverAnswers_should_giveUp() throws Exception {
// A broker under a memory or disk alarm accepts the connection then stops draining: the publish
// never returns on its own.
CountDownLatch release = new CountDownLatch(1);
when(connectionFactory.newConnection())
.thenAnswer(
invocation -> {
release.await(30, TimeUnit.SECONDS);
throw new IOException("released by the test");
});

long startedAt = System.nanoTime();
assertThrows(TimeoutException.class, () -> rabbitmqService.publish("http", "{}"));
long elapsedMs = (System.nanoTime() - startedAt) / 1_000_000;

// The caller holds the inject's database transaction: it must come back on its own.
assertTrue(elapsedMs < 10_000, "publish returned after " + elapsedMs + " ms");
release.countDown();
}
@RomuDeuxfois
Romuald Lemesle (RomuDeuxfois) force-pushed the fix/bound-rabbitmq-publish branch 5 times, most recently from 33ea5b8 to 5ac9ed6 Compare August 27, 2026 14:16
@RomuDeuxfois
Romuald Lemesle (RomuDeuxfois) marked this pull request as ready for review August 27, 2026 14:22
@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Container vulnerability scan — Passed

Previously reported findings are no longer present.

Image Critical High Total Status
Standard 0 0 0 ✅ Clear
UBI9 0 0 0 ✅ Clear

View workflow run · Standard JSON report · UBI9 JSON report

Updated from CI run attempt 1.

Executor.executeExternal publishes inside the inject's transaction. A
broker under a memory or disk alarm accepts the connection then stops
draining its sockets, so basicPublish blocks - and Java has no write
timeout to bound it. The job thread parks, the transaction stays open
with its row locks, the Hikari pool drains and the whole platform stops
answering, since the session store is in the same database.

Run the publish on an executor and give up after
openaev.rabbitmq.publish-timeout-ms. The worker stays parked - a socket
write is not interruptible - but the caller returns, its transaction
rolls back and the inject fails instead of the platform. The executor is
sized on the database pool, which already bounds how many publishes can
run at once.

Also drop the connection timeout from the client default of 60s to 10s:
publish opens a connection per call, so an unreachable broker would tie
a worker up for a full minute.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

filigran team Item from the Filigran team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants