[improve][build] Upgrade Shadow plugin to 9.6.1 and merge service descriptor files in shaded jars

[lhotari]

Motivation

Two related build changes to the Gradle Shadow plugin.

Shadow 9.4.2 → 9.6.1. Picks up the accumulated fixes since 9.4.2 — Zip Slip validation on ZIP entry names, relocation patterns included in the task fingerprint, afterEvaluate removed when adding variants — and moves the plugin's own log4j-core off 2.25.3.

Shaded jars are missing service providers. ShadowJar sets duplicatesStrategy = EXCLUDE in its own constructor, and Gradle applies the strategy before the resource transformers run. Every duplicate META-INF/services/* entry is dropped before ServiceFileTransformer sees it, so mergeServiceFiles() keeps only the first provider file it encounters instead of merging them all.

The jars built today are missing 86 service providers across 28 descriptor files in 6 jars:

Jar	Service descriptor	Providers today	Should be
jclouds-shaded	org.jclouds.apis.ApiMetadata	1	10
jclouds-shaded	org.jclouds.providers.ProviderMetadata	1	6
pulsar-client, pulsar-client-all, pulsar-client-admin	com.fasterxml.jackson.databind.Module	1	3–4
pulsar-client-all, pulsar-client-admin	org.glassfish.jersey.internal.spi.AutoDiscoverable	1	3
microbench	org.eclipse.jetty.ee10.webapp.Configuration	1	16

This is a regression introduced by the Gradle migration: the Maven-built 4.0.6 artifacts on Maven Central contain the fully merged descriptors, since maven-shade-plugin's ServicesResourceTransformer merged them correctly.

The practical impact is narrower than the numbers suggest, because Pulsar's own code does not rely on most of these lookups — jclouds ProviderMetadata instances are constructed directly in JCloudBlobStoreProvider, and PulsarAdminImpl registers MultiPartFeature explicitly. But ContextBuilder.newBuilder("transient") and any third-party ServiceLoader lookup against the published shaded jars resolve against truncated descriptors.

Shadow 9.5.0 added a warning for exactly this mismatch, which is how it surfaced during the upgrade.

Modifications

Upgrade (gradle/libs.versions.toml, build-logic/conventions/build.gradle.kts)

• shadow 9.4.2 → 9.6.1.
• Add the log4j-bom platform to the build-logic conventions dependencies, so the log4j Shadow drags onto the build classpath tracks the log4j2 catalog version instead of floating on whatever the plugin depends on.

Service descriptor fix (pulsar.shadow-conventions, microbench, pulsar-functions/runtime-all)

EXCLUDE stays the global strategy. Class files bypass the transformers entirely — ShadowCopyAction.visitFile handles them in a dedicated branch — so the strategy is the only thing deduplicating them, and jetty-upgrade/zookeeper-with-patched-admin depends on first-wins to override ZooKeeper's server.admin.* classes with its patched copies. Setting INCLUDE globally and deduplicating with PreserveFirstFoundResourceTransformer (as Shadow's docs suggest) does not work for that reason: it emits ~40 duplicated admin classes in that module.

Instead, only the paths a transformer owns are exempted:

mergeServiceFiles()
val transformedPaths = listOf("META-INF/services/**", "META-INF/*.kotlin_module")
filesMatching(transformedPaths) { duplicatesStrategy = DuplicatesStrategy.INCLUDE }
inputs.property("transformedPathsDuplicatesStrategy", "$transformedPaths=INCLUDE")
failOnDuplicateEntries.set(true)

• The inputs.property is required, not cosmetic: filesMatching {} actions are not part of a task's input fingerprint. Measured before adding it, :jclouds-shaded:shadowJar produced the identical build cache key af6dd43b… with and without the filesMatching block, and a --build-cache run served a pre-fix jar to the fixed configuration. Naming the patterns in an input property keeps the cache key honest across this change.
• META-INF/*.kotlin_module is covered for the same reason — KotlinModuleMetadataTransformer, which ShadowJar applies on its own, hits the identical problem. No duplicates exist there today, so this changes no jar content; it removes the warning and closes the latent hole.
• failOnDuplicateEntries guards against a future change letting an entry into the archive twice.

The three call sites are pulsar.shadow-conventions (which covers every module going through pulsar.client-shade-conventions and pulsar.minimized-dependencies-conventions) plus microbench and pulsar-functions/runtime-all, the two modules applying com.gradleup.shadow directly.

Verifying this change

• Make sure that the change passes the CI checks.

This change is a build-only change verified by comparing the produced artifacts:

• The upgrade produces byte-identical artifacts. Every one of the 9 shadowJar outputs was built under 9.4.2 and 9.6.1 with --rerun-tasks, and the entry lists plus per-entry CRCs are identical across all of them.
• The fix restores exactly the merged descriptors and nothing else. All 11 shaded jars were rebuilt and compared entry-by-entry against the pre-fix baseline: the only differences are META-INF/services/* contents, which now match a reference build that lets every duplicate through to the transformer.
• zookeeper-with-patched-admin still resolves to the patched classes. 45 of the 48 org.apache.zookeeper.server.admin.* classes in the shaded jar are this module's patched copies (verified by CRC against the module's build output), 0 are ZooKeeper's originals winning instead, all 48 ZK admin classes remain present, and the jar has 0 duplicate entries.
• failOnDuplicateEntries passes for every shaded jar, and no DuplicatesStrategy warnings remain in the build output.
• ./gradlew quickCheck and ./gradlew spotlessCheck checkstyleMain checkstyleTest pass.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

The Shadow Gradle plugin is upgraded from 9.4.2 to 9.6.1 (build-time only, not shipped). The service descriptor fix changes the contents of the published shaded jars — pulsar-client, pulsar-client-all, pulsar-client-admin, jclouds-shaded, pulsar-functions-local-runner-shaded — by restoring service providers that are currently dropped. No dependency versions change for the shipped artifacts.