Skip to content

dbconn: take RDS TLS from the driver, and drop rejectReadOnly - #1221

Merged
morgo merged 4 commits into
mainfrom
tls/retire-rds-autotls
Sep 7, 2026
Merged

dbconn: take RDS TLS from the driver, and drop rejectReadOnly#1221
morgo merged 4 commits into
mainfrom
tls/retire-rds-autotls

Conversation

@morgo

@morgo morgo commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

What

Bumps github.com/block/mysql to merged master and retires the RDS TLS machinery spirit and the driver now both have. +228 / −3076, the bulk of it rdsGlobalBundle.pem.

Deleted: the embedded bundle, the x509.CertPool built from it, and the local rdsAddr regex. NewTLSConfig and IsRDSHost stay as thin forwarders to mysql.RDSTLSConfig() / mysql.IsRDSAddr, so there is one bundle to refresh instead of two that can drift.

They stay rather than going away because two callers need them and neither is a database/sql connection the driver covers:

  • GetTLSConfigForBinlog builds a *tls.Config for the go-mysql binlog client — Vitess-lineage code, not the driver.
  • block/schemabot calls dbconn.IsRDSHost to pick a TLS mode (pkg/mysqlconn/mysqlconn.go:49).

initRDSTLS keeps registering under the name "rds" for the same reason: EnhanceDSNWithTLS returns DSNs carrying tls=rds, and schemabot opens them.

Two changes here are forced by the bump, not chosen

Both are behaviour. They are the reason this is not a pure deletion.

1. --tls-mode=DISABLED must now say tls=false out loud

The driver's auto-TLS fires when the DSN asked for nothing — which is exactly what cfg.TLSConfig = "" meant. So on an RDS host, DISABLED would have produced a TLS connection: the opposite of what the operator asked for, arriving from a dependency bump, with no error to notice.

Measured, with the old assignment restored:

mode=DISABLED  host=db.cxyz.us-east-1.rds.amazonaws.com:3306  tls_param=""  effective_tls=true
mode=DISABLED  host=mysql.internal:3306                       tls_param=""  effective_tls=false

It is RDS-only, so no test against a local MySQL could have caught it. And the tests that did cover DISABLED asserted the DSN string "should not contain tls=" — the spelling, not the effect. Seven such assertions across four files now go through a requireNoEffectiveTLS helper that parses the DSN and checks cfg.TLS == nil, and TestNewDSNDisabledMode covers an RDS address explicitly.

2. AllowCleartextPasswords had to grow a second condition in the same breath

It was cfg.TLSConfig != "". With fix 1 in place that reads the new "false" as "TLS is on" — and starts sending passwords in the clear over a plaintext connection. Fixing the first without the second would have been a straight security regression.

Each guard fails on its own mutation:

KILLED  DISABLED writes empty tls= (the original bug)
KILLED  cleartext guard drops the tlsDisabled check
KILLED  NewTLSConfig returns an empty root pool
KILLED  NewCustomTLSConfig ignores the empty-certData fallback

DBConfig.RejectReadOnly is deleted

Not a choice either: block/mysql#5 removed Config.RejectReadOnly and made the rejection unconditional, so conn.go:341 stopped compiling the moment the pin moved.

The interesting half is the opt-out. pkg/datasync/runner.go set RejectReadOnly = false for its read-only source, on the grounds that "sync's source is read-only by design, so it must not fire." That reasons from the wrong axis: 1290/1792/1836 are raised by writes, not by connecting to a read-only server. A pure reader cannot provoke them.

Verified rather than argued — a super_read_only MySQL 8.0, as the restricted source user (SELECT, REPLICATION SLAVE/CLIENT, RELOAD):

statement result
every session SET newDSN adds (sql_mode, time_zone, transaction_isolation, ...) OK
SHOW TABLES, SHOW CREATE TABLE, SELECT OK
SHOW MASTER STATUS OK
FLUSH BINARY LOGS (the one that looked risky) OK — and it genuinely rotated the binlog
INSERT — negative control ERROR 1290

So the probe can see a failure, and sync's actual workload does not trigger one. If sync ever does write to its source that is a bug, and the rejection now surfaces it instead of hiding it. The target side keeps the protection, which is where the Aurora-failover case actually lives.

Behaviour deltas from delegating the host match

  • Case-insensitive now. A fix: DNS is case-insensitive, nothing normalizes the host, and an uppercased hostname is the same endpoint.
  • GovCloud and China report false. The bundle holds no roots for either partition, so verifying against it could only ever fail. Reach them with --tls-certificate-path and that partition's own bundle.
  • MinVersion: tls.VersionTLS12 is now pinned; spirit's config left it to the Go default.

Not changed, but worth knowing

NewCustomTLSConfig(nil, ...) still falls back to the RDS roots for a non-RDS host with no --tls-certificate-path. That is inherited behaviour and it is close to useless — a non-RDS server will not present an RDS-issued certificate, so VERIFY_CA / VERIFY_IDENTITY fail by construction. Preserved deliberately: tightening it is a separate behaviour change and does not belong in a dependency bump. Flagged in the doc comment.

Verification

  • go build ./..., go vet ./..., gofmt -l, go mod tidy — all clean
  • go test ./... against MySQL 8.0.45: identical pass/fail set to origin/main, no new failures. pkg/dbconn fully green.
  • The 12 pre-existing failures (FK integration, minimal-RBR, transaction-compression) reproduce unchanged on a stashed origin/main tree — local compose configuration, not this branch. One pkg/status flake appeared once and passes 5/5 in isolation.

Related

Same retirement in the other two consumers: block/vitess#23, and strata's to follow. The driver side is block/mysql#4 and #5.

🤖 Generated with Claude Code

block/mysql now carries the RDS certificate bundle and applies verified TLS
to an RDS address by itself, so spirit's copy of the same machinery is
duplication with a second refresh schedule. rdsGlobalBundle.pem (2952 lines),
the pool built from it, and the local rdsAddr regex go; NewTLSConfig and
IsRDSHost stay as thin forwarders and keep their behaviour.

They stay because two callers still need them, and neither is a database/sql
connection the driver would cover: GetTLSConfigForBinlog builds a config for
the go-mysql binlog client, and block/schemabot calls IsRDSHost to pick a TLS
mode. initRDSTLS keeps registering under "rds" for the same reason —
EnhanceDSNWithTLS returns DSNs naming it, and schemabot opens those.

Delegating changes two things. The host match is now case-insensitive, which
is a fix: DNS is case-insensitive, nothing normalizes the host, and an
uppercased hostname is the same endpoint. GovCloud and China endpoints now
report false, because the bundle holds no roots for either partition and
verifying against it could only ever fail. Also new for free: MinVersion is
pinned to TLS 1.2, which spirit's config did not set.

Two changes here are forced by the pin rather than chosen, and both are
behaviour, not cleanup.

--tls-mode=DISABLED now writes tls=false instead of leaving tls= empty. An
empty tls= is precisely what the driver's auto-TLS treats as "nothing was
asked for", so on an RDS host DISABLED would have produced a TLS connection —
the opposite of what was requested, from a dependency bump, with no error.
Measured, with the old assignment restored:

    mode=DISABLED  host=db.cxyz.us-east-1.rds.amazonaws.com  effective_tls=true
    mode=DISABLED  host=mysql.internal                       effective_tls=false

It is RDS-only, so no test against a local MySQL could have seen it — and the
tests that covered DISABLED asserted the DSN string "should not contain tls=",
which is the spelling rather than the effect. Those now assert cfg.TLS is nil,
via a requireNoEffectiveTLS helper, and TestNewDSNDisabledMode covers an RDS
address explicitly.

AllowCleartextPasswords had to grow a second condition in the same breath. It
was `cfg.TLSConfig != ""`, which reads the new "false" as "TLS is on" and
would send passwords in the clear over a plaintext connection. Both guards
are load-bearing and each fails on its own mutation:

    KILLED  DISABLED writes empty tls= (the original bug)
    KILLED  cleartext guard drops the tlsDisabled check
    KILLED  NewTLSConfig returns an empty root pool
    KILLED  NewCustomTLSConfig ignores the empty-certData fallback

DBConfig.RejectReadOnly is deleted because the driver deleted the option it
mapped to: read-only errors now always recycle the connection. The field also
carried an opt-out that the sync runner set for its read-only source, and that
turned out to be guarding against an error the workload cannot raise —
1290/1792/1836 come from *writes*, not from connecting to a read-only server.
Verified against a super_read_only MySQL 8.0 as the restricted source user:
SELECT, SHOW TABLES, SHOW CREATE TABLE, SHOW MASTER STATUS, every session SET
newDSN adds, and the binlog client's FLUSH BINARY LOGS all succeed; INSERT is
what returns 1290. If sync ever does write to its source, that is a bug, and
the rejection now surfaces it rather than hiding it.

Full suite run against MySQL 8.0.45: identical pass/fail set to origin/main,
no new failures. pkg/dbconn green.
@aparajon

aparajon commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

🤖 Adversarial correctness reviewf224250b (+228/-3076, 11 files)

The two forced behaviour changes are correctly identified and correctly paired. --tls-mode=DISABLED leaving TLSConfig empty would have handed RDS users TLS from a dependency bump, and the AllowCleartextPasswords guard genuinely had to move in the same commit — != "" reading "false" as "TLS is on" is a security regression, not a cosmetic one. I re-ran both mutations independently and both die:

KILLED  cfg.TLSConfig = tlsDisabledConfigName  →  ""
KILLED  cleartext guard drops the tlsDisabled check
KILLED  NewCustomTLSConfig's empty-certData fallback → x509.NewCertPool()

The RejectReadOnly reasoning is right too, and for the reason given rather than by luck: 1290/1792/1836 come from writes, and the guard was protecting a reader from an error a reader cannot provoke. Retiring it while keeping the target side's protection is the correct shape.

Two findings, both about what the retained surface promises callers.

# Sev Where What
1 med conn.go:208-217 The "rds" registration is kept because "block/schemabot opens them" — but after the block/mysql switch, schemabot opening them is precisely what fails. Measured: invalid value / unknown config name: rds, raised at the ParseDSN schemabot calls, before sql.Open
2 low-med conn.go:513, :556 EnhanceDSNWithTLS is the other DSN producer and did not get fix 1. For DISABLED it returns a DSN with no tls= at all — the exact input the driver's auto-TLS overrides. Spirit's own caller survives only because newDSN runs afterwards

1 — the registration's stated beneficiary is the consumer it breaks (med)

// The registration survives the driver's own auto-TLS because the name is part
// of this package's contract, not an internal detail: EnhanceDSNWithTLS returns
// DSNs carrying tls=rds, and block/schemabot opens them.

The first half is right and the retention is right. The second half is the part I'd change, because it reads as reassurance and the situation is the opposite.

Measured, with a scratch module importing this branch's pkg/dbconn alongside upstream go-sql-driver/mysql — the pairing schemabot has today:

enhanced          = "u:p@tcp(db.cxyz.us-east-1.rds.amazonaws.com:3306)/app?tls=rds"
upstream.ParseDSN → err = invalid value / unknown config name: rds   (cfg == nil)
sql.Open("mysql") → err = invalid value / unknown config name: rds

It fails at ParseDSN, which matters for where the error surfaces. pkg/mysqlconn/mysqlconn.go in schemabot does exactly this sequence — dbconn.EnhanceDSNWithTLS at :157, then mysql.ParseDSN at :163 with upstream imported at the top of the file — so the failure arrives as parse enhanced DSN: invalid value / unknown config name: rds, on every RDS MySQL connection, before any dial. Its tlsModeForHost returns REQUIRED for exactly the hosts IsRDSHost matches, so the RDS path is the only one affected and it is affected completely.

To be clear about blame: this is not introduced here. Spirit's move to github.com/block/mysql did it, and schemabot is fine today only because its pin (v0.16.1-0.20260903162727-fc5f1dfb0a40) still imports upstream in pkg/dbconn/conn.go. The break lands the moment schemabot bumps — which the block/mysql wave makes imminent, and which nothing in either repo will catch at compile time. The DriverName doc comment two dozen lines up already states the hazard precisely; it is only this comment, and the PR summary, that phrase it as a reason the registration still works.

So: keep the registration, and say what actually has to happen. Something like "the name is what EnhanceDSNWithTLS's DSNs carry, so a consumer must open them with DriverName — block/schemabot currently opens with upstream go-sql-driver and will need to switch when it bumps past this." That turns a line a schemabot reader would take as an all-clear into the one-line migration note they need.

The alternative worth considering, since EnhanceDSNWithTLS exists mainly to hand a DSN to someone else: export the *tls.Config (or a func(*mysql.Config)) so a consumer can apply TLS without routing through a registry name that only one driver package can resolve. That removes the class rather than documenting it, but it is a bigger change than this PR and I would not hold the bump for it.

2 — the DISABLED fix landed on one of the two DSN producers (low-med)

newDSN now writes tls=false. addTLSParametersToDSN, reached through the exported EnhanceDSNWithTLS, still has:

case "DISABLED":
	return dsn, nil // No TLS needed

and EnhanceDSNWithTLS short-circuits above it with return inputDSN, nil for the same mode. Both hand back a DSN with no tls= — which is the exact condition fix 1 identifies as unsafe, since that is what the driver's auto-TLS treats as "the DSN asked for nothing."

Spirit's one internal caller is safe, and I checked rather than assumed: runner.go:1276 passes the enhanced replica DSN straight into NewWithConnectionType, which calls newDSN, which applies the fix. So there is no live bug in this repo.

But the export is documented as producing a DSN to open, the reason it exists is to hand one to a caller that will open it themselves, and for DISABLED that caller gets TLS on RDS. Making the two producers agree is small — return cfg.TLSConfig = tlsDisabledConfigName from the DISABLED branch instead of the untouched DSN — and it keeps the invariant in one place rather than depending on which entry point a consumer happened to use. If the asymmetry is deliberate (enhance means "add TLS, never remove it"), that is a reasonable position, but then it belongs in the doc comment, because the mode's name promises the opposite.


Verified — the delegation, and three attacks that dissolved

The delegation is sound at the pinned version. mysql.IsRDSAddr is rdsAddr.MatchString(addr) && !govCloudAddr.MatchString(addr), both (?i), so the case-insensitivity and GovCloud/China claims in the summary hold exactly as written. rdsRootCAs is a sync.OnceValue the driver never hands out directly and RDSTLSConfig clones — so NewCustomTLSConfig's "a private copy of the pool" comment is true rather than hopeful.

Attack that dissolved: the private-pool claim resting on an untested dependency property. TestValidCertificateBundle already pins it from spirit's side — require.NotSame(cfg.RootCAs, other.RootCAs) plus a non-empty check on the NewCustomTLSConfig(nil, …) fallback. If the driver ever regressed to sharing, this test fails here rather than silently widening trust. Good test to have written.

Attack that dissolved: --tls-mode=DISABLED on the replica path. replicaDBConfig.TLSMode does inherit DISABLED from the main config, and EnhanceDSNWithTLS returns the DSN untouched — but NewWithConnectionType runs newDSN on it immediately after, so the replica gets tls=false like everything else. Finding 2 is the exported contract only.

Attack that dissolved: the canRetryError comment rewrap. The reflowed line runs long, but pkg/dbconn/dbconn.go already carries pre-existing comment lines at 110, 141, 170 and 222 characters, so this is house style rather than a regression.

Local. gofmt -l, go build ./..., go vet ./pkg/dbconn/ ./pkg/datasync/ clean. go test ./pkg/dbconn/ green on the TLS/DSN subset that runs without a server (the rest of the package needs MySQL, so I did not use full-package runs to judge mutations). All mutations restored from backup; the worktree is clean at f224250b.

Leak check: clean. The probe hostname is invented, and the schemabot paths cited are public.

This review was generated by Claude Code (claude-opus-5).

@aparajon aparajon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Approving — the two forced behaviour changes are correctly identified and correctly paired, and I re-ran the mutations that prove it. Findings are in the review comment above; the one worth acting on before the bump propagates is finding 1 — the tls=rds registration is retained on the grounds that schemabot opens those DSNs, and after the block/mysql switch that is exactly the case that fails (invalid value / unknown config name: rds, at the ParseDSN schemabot calls).

This stamp was left by Claude Code (claude-opus-5).

@Kiran01bm

Copy link
Copy Markdown
Collaborator

🤖 Review findings - created by Kiran's code review agent - for spirit/pull/1221, f224250.

Verdict: 8 findings — 0 blocking, 6 non-blocking (one missed downstream symbol, and four unasserted security properties on paths this PR rewired), 2 suggestions. The DISABLED-mode fix is real and its new tests are individually load-bearing.

Non-blocking

1. The PR keeps three symbols for schemabot's sake and misses a fourth. GetEmbeddedRDSBundle is deleted, but block/schemabot calls it at pkg/engine/postgres/postgresconn.go:402 — and the PR body explicitly preserves NewTLSConfig, NewCustomTLSConfig and IsRDSHost because schemabot uses them. Low blast radius (schemabot pins a released version and the call site's migration is nil), but the consumer note should name it so the bump isn't a surprise compile break.

2. The binlog TLS path — the stated reason NewTLSConfig survives — has no verification-strength assertion. Rewriting conn.go:683's VERIFY_IDENTITY branch to build a PREFERRED config (InsecureSkipVerify: true) passes the whole suite. The branch is under test — deleting tlsConfig.ServerName = host at :708 is caught by every enabled_* subtest of TestGetTLSConfigForBinlogCaseInsensitive — so the tests reach it and assert only ServerName. A regression here drops verification on the replication stream while the database/sql pool stays correctly verified, so nothing looks wrong. One require.False(cfg.InsecureSkipVerify) per mode closes it.

3. initCustomTLS can register a skip-verify config under the verify_identity name undetected. Making conn.go:235 register NewCustomTLSConfig(certData, "PREFERRED") for all three strict names survives every test: the DSN still spells tls=verify_identity, the connection still succeeds, and zero certificate verification happens. This is the exact function whose certData sourcing the PR rewired, so it is the one place worth pinning by strength rather than by name.

4. A 0-byte --tls-certificate-path file silently falls back to Amazon's RDS roots. conn.go:126 treats empty certData as "use the driver's roots", which is correct for the nil call but indistinguishable from an operator pointing at a truncated or empty CA file for a private CA. That user asked for VERIFY_IDENTITY against their own root and gets Amazon's instead, with no error. Erroring on a non-nil-but-empty read would separate the two intents.

5. The deleted bundle's per-certificate validation is not replaced. The old TestValidCertificateBundle looped every PEM block in rdsGlobalBundle and required x509.ParseCertificate to succeed; the rewrite asserts only that the pool is non-empty, per-call, and not skip-verify. Spirit will now catch an entirely empty root pool but not a truncated or wrong-partition bundle arriving via a future block/mysql bump — which surfaces at connect time in production as an unknown authority error naming nothing in this repo. Narrow loss (the old test never proved the roots were right), but asserting a known subject such as Amazon RDS Root 2019 CA would restore it cheaply.

6. The datasync source-config relaxation this PR rewrites is unpinned. Deleting r.sourceDBConfig.ForceKill = false at datasync/runner.go:265 passes all 41 datasync tests, E2E included, because the local test MySQL grants tsandbox the kill privilege — so the least-privilege path is never exercised. The PR's central claim (everything sync sends the source succeeds against a super_read_only MySQL) is verified in the commit message but has no regression test; sync_test.go:169-170 asserts only MaxOpenConnections on these configs.

General suggestions

7. VERIFY_CA's hand-rolled chain check is never invoked by a test. conn.go:152's VerifyPeerCertificate closure can be replaced wholesale with return nil and the suite stays green — tests assert the struct shape (InsecureSkipVerify == true, RootCAs != nil) but never the callback's behaviour. Since VERIFY_CA is implemented as skip-verify plus that callback, a regression there degrades it to no verification at all, strictly worse than PREFERRED. Pre-existing, but this PR changes which pool the closure captures.

8. These security assertions only ever run inside the Docker+MySQL matrix. The new TLS/DSN tests need no database — they pass standalone — but there is no unit-only job: linter.yml is golangci-lint only, and every go test lives in mysql*-docker.yml. That workflow also has a documentation-only short-circuit, so a PR classified as docs-only skips them entirely. A cheap always-on go test ./pkg/dbconn/... -run 'TLS|DSN|Certificate|RDS' job would gate the highest-value assertions in this PR.

The one thing that could have broken, verified

The whole PR rests on applyRDSAutoTLS being genuinely new in the block/mysql pin — if it weren't, deleting the embedded bundle would remove RDS roots with nothing replacing them. Verified against the pinned module source in GOMODCACHE: the driver hook is new, and restoring the pre-PR cfg.TLSConfig = "" for --tls-mode=DISABLED is killed by exactly one test — the RDS subtest of TestNewDSNDisabledMode, not the 127.0.0.1 one. That is the bug: before this PR, DISABLED against an RDS host had TLS silently forced back on by the driver.

Verified correct

  • Each new assertion in TestValidCertificateBundle is individually load-bearing: non-empty pool, InsecureSkipVerify == false, and per-call pool distinctness each kill a distinct mutant alone.
  • The empty-certData fallback is pinned — building an empty pool instead of the driver's roots is caught, and notably pkg/change/tls_test.go (which now passes nil) does not catch it.
  • requireNoEffectiveTLS is a genuine strengthening over the old NotContains(dsn, "tls="): it catches both tls=skip-verify smuggled in as "disabled" and the dropped half of the cleartext guard.
  • Removing require.True(cfg.RejectReadOnly) and the rejectReadOnly=true DSN param is forced by the driver field's deletion, not a weakening — the code would not compile otherwise.
  • IsRDSHost is pinned in both directions (five tests catch always-false; always-true breaks the pooled integration tests).
  • The PREFERRED default is pinned by four independent tests.
  • No //go:build tag and no t.Skip in any of the five modified test files, so nothing new is silently excluded.

This review was generated by Claude Code (claude-opus-5).

…S strength

Two behaviour fixes and the test coverage the review found missing.

EnhanceDSNWithTLS and addTLSParametersToDSN both returned an untouched DSN for
TLSMode=DISABLED. That is the same bug already fixed in newDSN: a DSN carrying
no tls= at all is exactly what the driver reads as permission to apply RDS
auto-TLS, so DISABLED handed RDS callers the opposite of what they asked for.
Both now write tls=false. An explicit tls= in the DSN still outranks the mode,
and a nil config is still a no-op — "nothing was said about TLS" is not the
same as "no TLS", and only the latter should write anything.

initCustomTLS now rejects a configured-but-empty certificate file.
NewCustomTLSConfig reads zero bytes as "use the RDS roots", which is correct
for a caller that named no path and wrong for one that named a path to a
truncated or unpopulated private CA: that operator asked to verify against
their own root and would have silently verified against Amazon's, on a
connection that succeeded. Only the path distinguishes the two intents, so the
check has to live at the read site.

New tls_strength_test.go asserts verification strength rather than struct
shape, which is what every weakening in this package preserves:

- VERIFY_CA is InsecureSkipVerify plus a VerifyPeerCertificate callback, so
  the callback is the only thing verifying anything. It is now exercised
  against real certificates — a leaf from the configured CA is accepted
  despite a mismatched hostname, a leaf from an unknown CA is rejected. The
  reviewer's `return nil` mutant degraded the mode to no verification at all
  under a name promising the opposite; it now dies.
- InsecureSkipVerify is pinned false for VERIFY_IDENTITY, NewTLSConfig, and
  the binlog path. That path is the stated reason NewTLSConfig survives the
  retirement and is separate from the database/sql pool, so a weakening there
  drops verification on the replication stream while the pool stays correct.
- A config registered under a mode's name is read back through ParseDSN and
  checked for that mode's strength, closing the gap where initCustomTLS could
  register skip-verify as "verify_identity" undetected.
- The RDS pool is checked for a known Amazon root, restoring what the deleted
  bundle's per-certificate test used to cover: spirit no longer owns the
  bundle, so what matters is that a future block/mysql bump cannot supply a
  pool that is non-empty but not the RDS trust store.

Mutations, all killed:

    KILLED  VERIFY_CA callback body -> return nil
    KILLED  VERIFY_IDENTITY InsecureSkipVerify false -> true
    KILLED  empty-certificate-file check removed
    KILLED  EnhanceDSNWithTLS DISABLED -> untouched DSN

Three existing DISABLED assertions were asserting the DSN spelling rather than
the effect ("should not contain tls="), which is what made the original bug
invisible. The two table rows now pin tls=false and the replica test asserts no
effective TLS.

Added .github/workflows/unit-tls.yml. Every go test in this repo lived in the
mysql*-docker matrices, so these assertions were gated behind Docker, a MySQL
boot, and each workflow's documentation-only short-circuit. They need no
database and now run on every pull request on their own.

The initRDSTLS comment no longer cites schemabot as evidence the tls=rds
registration works, since after the block/mysql switch schemabot is precisely
what would break. It now states the requirement — open those DSNs with
DriverName — and notes schemabot is switching alongside this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@morgo

morgo commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator Author

🤖 Addressed. Both reviews converged on the same weak spot from different directions, and it turned out to be a real bug rather than a doc nit.

@aparajon finding 2 / @Kiran01bm's framing — fixed, and it was the same bug twice. EnhanceDSNWithTLS and addTLSParametersToDSN both returned an untouched DSN for DISABLED, which is exactly the condition the driver reads as "apply RDS auto-TLS." Both now write tls=false. Precedence is preserved in both directions, which I checked rather than assumed: an explicit tls= in the DSN still outranks the mode, and a nil config is still a no-op — "nothing was said about TLS" is not "no TLS", and only the second should write anything.

The three tests that broke were all asserting the DSN spelling (should not contain "tls=", returns original DSN unchanged). That is precisely why the original bug was invisible, so the same treatment as last round: the table rows pin tls=false, the replica test asserts no effective TLS.

@aparajon finding 1 — comment rewritten, and you were right that it read as an all-clear. It no longer cites schemabot as evidence the registration works, because after the switch schemabot is the thing that breaks. It now states the requirement (open those DSNs with DriverName) and notes schemabot is switching to block/mysql alongside this. I confirmed your measurement independently, and the reason it resolves is that once schemabot imports block/mysql it shares that driver's registry — so tls=rds resolves again. Same-registry, not same-name.

@Kiran01bm 7 — best finding in either review, fixed. VERIFY_CA is InsecureSkipVerify plus the callback, so the callback is the only thing verifying anything, and return nil left the struct identical while degrading the mode below PREFERRED. It is now exercised against real generated certificates: a leaf from the configured CA is accepted despite a deliberately mismatched hostname (that is the mode's whole point), and a leaf from an unknown CA is rejected.

@Kiran01bm 2, 3, 5 — fixed, all as strength assertions rather than shape:

# now pinned by
2 InsecureSkipVerify false on the binlog path for REQUIRED and VERIFY_IDENTITY
3 the registered config read back through ParseDSN and checked for its mode's strength
5 the RDS pool checked for Amazon RDS Root 2019 CA, so a future bump cannot hand over a non-empty pool that is not the RDS trust store

@Kiran01bm 4 — fixed, and it needed to move. A configured-but-empty CA file now errors. The check has to sit at the read site in initCustomTLS, because by the time NewCustomTLSConfig sees len(certData) == 0 the two intents are indistinguishable — only the path separates "named no CA" from "named a CA file that is empty."

@Kiran01bm 8 — fixed. Added .github/workflows/unit-tls.yml. You were right that every go test lived in the mysql*-docker matrices behind a docs-only short-circuit. I verified the subset is genuinely database-free (every matched test is a pure DSN/TLS-config function, whole set runs in under a second) and it now runs on every PR with nothing to provision.

Mutations, each killed:

KILLED  VERIFY_CA callback body -> return nil
KILLED  VERIFY_IDENTITY InsecureSkipVerify false -> true
KILLED  empty-certificate-file check removed
KILLED  EnhanceDSNWithTLS DISABLED -> untouched DSN

Not doing two, deliberately:

  • @Kiran01bm 1 (GetEmbeddedRDSBundle) — real break, but it is being fixed rather than accommodated. The one call site is schemabot's Postgres trust pool, which wants an *x509.CertPool; it now takes mysql.RDSTLSConfig().RootCAs directly, so no byte-slice accessor has to survive anywhere. That is in the companion schemabot PR, which also moves it to block-mysql and handles the fact that the hot-swap driver keeps upstream go-sql-driver linked.
  • @Kiran01bm 6 (datasync ForceKill) — genuine coverage gap and correctly diagnosed (the local test MySQL grants tsandbox the kill privilege, so the least-privilege path never runs). Pinning it needs a restricted user in the test fixture, which is a fixture change rather than a TLS change, and the field is untouched by this PR. Leaving it out rather than half-testing it.

Full pkg/dbconn, pkg/datasync and pkg/change green locally.

@morgo
morgo enabled auto-merge (squash) September 7, 2026 00:27
Comment thread .github/workflows/unit-tls.yml Fixed
zizmor/cache-poisoning: the job also runs on release tags, where a restored
module cache could have been written from a lower-trust context. linter.yml
disables the cache for the same reason. The job compiles a single package, so
the cache was buying almost nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TestReplicaTLSEnhancement was the fourth test asserting how a DISABLED DSN is
spelled rather than what it does, and the one the new TLS/DSN job could not
catch: that job runs pkg/dbconn only, because pkg/migration's TestMain
provisions a server.

Its table carried a shouldEnhance bool covering two properties that DISABLED
has now pulled apart — "returned byte-identical" and "no TLS results". DISABLED
writes tls=false, so its text changes while its effect does not; the row asserting
equality had to fail. A bool cannot express both, so it is now a three-valued
expectation and each row says which property it means. The row that preserves a
DSN's own tls=skip-verify still demands byte-identity, because there the text
*is* the contract.

RequireNoEffectiveTLS moves to pkg/testutils, which both packages already
import, and pkg/dbconn keeps a one-line local forwarder so its six call sites
are untouched. Two copies of "what counts as no TLS" would let one be
strengthened while the other silently stayed weak — the same trap the assertions
above fell into.

Also dropped expectedReplicaTLSMode from the table: nothing read it. It was
formatted like an assertion and was not one.

The workflow comment now says it covers pkg/dbconn only, so a green tick there
is not mistaken for the whole TLS surface.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@aparajon

aparajon commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

🤖 Re-reviewf224250bff47088d (approval stands)

Both findings are addressed, and the two extras you found on the way are the better half of the commit. I re-ran your mutation table against the workflow's own filter rather than the package, so the result also answers "does the new job actually protect this":

KILLED  VERIFY_CA callback -> return nil          TestVerifyCAActuallyVerifiesTheChain
                                                  TestVerifyCARejectsRDSChainWhenPrivateCAConfigured
KILLED  EnhanceDSNWithTLS DISABLED -> untouched   TestEnhanceDSNWithTLS
                                                  TestDisabledModeProducesNoTLSFromEitherDSNProducer
KILLED  empty-certificate-file check removed      TestEmptyCertificateFileIsAnError

Finding 1 is closed properly rather than reworded — the comment now states the requirement, and block/schemabot#1320 makes the "switching alongside this change" clause true instead of aspirational. Finding 2 landed on both producers, and separating "the caller said nothing about TLS" (nil config, leave it alone) from "the caller asked for none" (DISABLED, say tls=false positively) is the right cut; the DSN-outranks-mode ordering is preserved, and tls=false is a value upstream go-sql-driver parses too, so the fix does not widen the blast radius of the one remaining incompatible value.

The empty-CA-file catch is the one I'd have missed. "Named a path to a truncated private CA" and "named no path" are genuinely only distinguishable at the read site, and silently verifying against Amazon's roots on a connection that succeeds is the worst shape that class of bug comes in.

One finding on the new workflow.

# Sev Where What
1 low .github/workflows/unit-tls.yml:47 The -run filter is case-sensitive, and this package names TLS modes in ALL CAPS — so Verify misses VERIFY_CA and Disabled misses DISABLED. Two pure tests are dropped, one of them covering DISABLED DSN behaviour

1 — the filter misses the repo's own naming convention (low)

The comment above it is right about the failure mode and calls the shot: "a new TLS test whose name this misses is silently unprotected." That case already exists, because the filter's vocabulary is title-case and the mode names are not:

$ go test -list 'TLS|DSN|Certificate|RDS|Verify|Strict|Disabled|Registered' ./pkg/dbconn/   → 44
$ go test -list '(?i)TLS|DSN|Certificate|RDS|Verify|Strict|Disabled|Registered' ./pkg/dbconn/ → 46

gained by (?i):
  TestVERIFY_CAHostnameFlexibility
  TestPREFERREDModeDISABLEDFallback

VERIFY_CA does not contain Verify; DISABLED does not contain Disabled. Both gained tests are pure newDSN assertions needing no server, and TestPREFERREDModeDISABLEDFallback covers DISABLED DSN construction — the behaviour this PR changes.

I'm calling it low rather than medium because neither test is uniquely protecting anything today: I could not construct a mutation that survives the current filter but dies under (?i), since TestDisabledModeProducesNoTLSFromEitherDSNProducer already covers the DISABLED path from both producers. The cost is future coverage, which is exactly what the comment is trying to buy. (?i) at the front of the pattern is the whole fix and it is free — I ran it with -race and no database:

go test -race -count=1 -run 'TLS|DSN|…'      ok  github.com/block/spirit/pkg/dbconn  2.638s
go test -race -count=1 -run '(?i)TLS|DSN|…'  ok  github.com/block/spirit/pkg/dbconn  2.112s

Residual after that: TestPREFERREDModeConfigConsistency is pure and still missed, because it carries no word from the vocabulary at all. Adding Preferred|Mode would pull it in along with the rest of tls_mode_test.go, which by the comment's own logic is the safe direction.

Also verified

The cache commit is a real fix, not a workaround. I looked at it expecting a test-masking change and it isn't one: cache: false on setup-go for a job that also runs on release tags, matching linter.yml, on a job that compiles one package. Nothing is skipped.

Three DISABLED assertions moved from spelling to effect. That was the actual reason the original bug was invisible — "should not contain tls=" is satisfied by exactly the DSN that triggers auto-TLS. Asserting tls=false and "no effective TLS" is the axis that can fail.

The RDS-pool test earns its place for the reason given. Spirit no longer owns the bundle, so "non-empty" is not the property worth pinning — "is actually the Amazon trust store" is, and that is what a bad block/mysql bump would break.

gofmt -l and go vet ./pkg/dbconn/ clean at ff47088d. All mutations restored from backup; worktree clean.

This review was generated by Claude Code (claude-opus-5).

@aparajon aparajon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Re-approving on ff47088d so the approval points at what is actually there — the previous one was on f224250b.

Both findings from the first round are closed, and I re-ran your mutation table against the new workflow's own filter rather than the package, so all three kills also confirm the job protects them. One low finding on that filter in the re-review: it is case-sensitive, so Verify misses VERIFY_CA and Disabled misses DISABLED, dropping two pure tests. (?i) recovers both and costs nothing.

This stamp was left by Claude Code (claude-opus-5).

@morgo
morgo merged commit 10804bb into main Sep 7, 2026
19 checks passed
morgo added a commit to block/schemabot that referenced this pull request Sep 7, 2026
Both blockers landed, in both modules:

  github.com/block/spirit  → 10804bbe (block/spirit#1221 merge commit)
  vitess.io/vitess         → 88d15fda (block/vitess#23 merge commit, release-24.0)

`go get github.com/block/spirit@main` resolved to an older revision than the
branch pin it replaced — the proxy had not indexed the merge yet — so both are
pinned to their merge commits explicitly.

The consumer module's vitess replace is byte-identical to the parent's again,
which is the invariant its own comment states and which was the source of the
CI failure before this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
morgo added a commit to block/schemabot that referenced this pull request Sep 7, 2026
* feat: use block/mysql (driver name "block-mysql")

block/mysql is now a hard fork with its own module path rather than a
replace-target for go-sql-driver/mysql, so SchemaBot imports it directly and
opens pools under the name it registers, "block-mysql". This also unblocks the
spirit bump: spirit's pkg/dbconn moved to block/mysql and retired its own copy
of the RDS certificate bundle, and a tls=rds DSN from EnhanceDSNWithTLS only
resolves for a consumer using the same driver package's TLS registry.

Both MySQL drivers stay linked, and that is not incidental:
go-mysql/hotswap-dsn-driver embeds upstream go-sql-driver and cannot be pointed
at the fork, so the credential-reloading storage pool keeps returning upstream's
*mysql.MySQLError while every pool SchemaBot opens itself returns the fork's.
The names differ, so registration does not collide -- verified: sql.Drivers()
reports all three of block-mysql, mysql and mysql-hotswap-dsn.

What does not survive that split is errors.As. The two MySQLError structs are
field-identical but live in different packages, so asserting one type silently
returns false for the other -- and silently is the problem. A retry classifier
checking only one type does not fail loudly; it stops recognizing deadlocks and
starts surfacing them as permanent errors. So error codes are now read through
mysqlerr.Number/Is, which accepts either, and no call site asserts a driver's
error type:

- pkg/storage/internal/sqlstore/error_classifier.go -- the storage pool, which
  is exactly the pool that can be opened either way
- pkg/mysqlerr.Reason
- pkg/engine/spirit.isLockWaitTimeout

pkg/mysqlerr/number_test.go pins both directions, including a test asserting
the two types are *not* interchangeable, so if a future dependency change ever
merges them the second branch is reported as dead rather than left looking
like superstition.

postgresconn's rdsRootPool took spirit's deleted GetEmbeddedRDSBundle. It now
takes mysql.RDSTLSConfig().RootCAs, which is the pool it wanted anyway: the
roots are the same because RDS issues from the same private Amazon CAs
regardless of engine, and RDSTLSConfig clones per call so nothing is aliased
with the MySQL side.

TestOpenNormalizesRDSDSNBeforeOpening asserted the driver name "mysql", which
now belongs to upstream -- opening under it would silently bypass the fork
rather than fail, so the assertion is worth keeping rather than deleting.

The sadscan annotations cover pre-existing false positives -- a doc-comment URI
shape with literal user:pass placeholders, obviously-fake test fixtures, and
three prose comments containing the word "PlanetScale" -- that this change put
into a commit for the first time by touching those files' imports. Each was
verified byte-identical on main.

Verified: go build ./..., go vet ./... and gofmt clean; go test ./pkg/...
fully green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Tidy the consumer module for the driver switch

Every red check traced to one cause: e2e/consumermodule is a second module,
and its go.mod was not regenerated after the parent's dependency graph changed.
CI runs `go test -race -run '^$' ./...` there and the toolchain refused with
"updates to go.mod needed". Unit Tests got through five minutes of real tests
before dying on the same step; Lint was cancelled behind the Build failure
rather than finding anything.

Its vitess replace was also pinned to an older block/vitess SHA than the
parent's, which the comment directly above it says must not happen ("Mirror the
parent module's replace directives; replaces do not propagate across module
boundaries"). Now matching, with block/mysql picked up and spirit moved onto the
same pin as the parent.

go-sql-driver/mysql stays in the graph as an indirect dependency, which is the
intended end state, not leftover: hotswap-dsn-driver embeds it, so both drivers
link, and pkg/mysqlerr is the only package importing it directly — for exactly
that reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Repoint spirit and vitess at merged revisions

Both blockers landed, in both modules:

  github.com/block/spirit  → 10804bbe (block/spirit#1221 merge commit)
  vitess.io/vitess         → 88d15fda (block/vitess#23 merge commit, release-24.0)

`go get github.com/block/spirit@main` resolved to an older revision than the
branch pin it replaced — the proxy had not indexed the merge yet — so both are
pinned to their merge commits explicitly.

The consumer module's vitess replace is byte-identical to the parent's again,
which is the invariant its own comment states and which was the source of the
CI failure before this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* mysqlerr: use slices.Contains in Is

golangci-lint's modernize check, on the new file. This is the first run where
lint actually got to execute — the earlier ones were cancelled behind the
consumer-module build failure, so it had never linted this code.

Full-repo golangci-lint v2 is clean locally.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Fix two driver names my sweep missed, and one that was wrong before it

Neither site was an inline sql.Open("mysql", …) literal, which is what the
original sweep grepped for, so both survived it:

  pkg/testutil/mysql.go   — wait.ForSQL's driver-name argument
  pkg/namedlock/…_test.go — a driver name carried in a table field

The testutil one is what failed CI: it runs inside a testcontainers start hook,
so `unknown driver "mysql"` surfaced as `FAIL github.com/…/pkg/namedlock`
rather than as a bad driver name, before any test ran. It is now a named
constant next to the blank import it has to agree with.

Worth recording why only one of the six packages using that helper failed. Only
pkg/namedlock does not link upstream go-sql-driver:

  pkg/namedlock                   upstream linked = 0   ← failed
  pkg/pendingdrops                upstream linked = 1
  pkg/engine/spirit               upstream linked = 1
  pkg/storage/internal/sqlstore   upstream linked = 1

Everywhere else upstream's init registers "mysql", so the readiness probe
resolved and passed — using upstream's driver, not the fork the blank import
declares. That was true before this PR too. So this is not only a fix for the
red package; it is the point at which all six actually probe with block/mysql.

Verified with real containers: namedlock 10.1s, pendingdrops 7.4s,
sqlstore 22.0s, all ok under -tags=integration. golangci-lint clean on all
three tag variants CI runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Drop the tidb parser replace

Spirit no longer needs the fork — it carries its own pkg/parser in-tree, and
its go.mod requires upstream github.com/pingcap/tidb/pkg/parser as a plain
indirect with no replace of its own. So the redirect this repo carried "for
SPATIAL index support in Spirit v0.13.0" no longer redirects anything anyone
reaches: `go mod why` now answers "main module does not need package
github.com/pingcap/tidb/pkg/parser".

Removed from both modules, since the consumer module mirrors the parent's
replaces. The upstream indirect requirement stays; it is only the fork
redirection that goes.

Verified: build clean, golangci-lint clean on all three tag variants, and
pkg/engine/spirit green under -tags=integration with real containers (20.2s) —
that being the package that would notice a parser regression.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Reimplement DSN credential reload in-repo; drop go-sql-driver/mysql

Retires github.com/go-mysql/hotswap-dsn-driver, the last thing in
SchemaBot's dependency graph that reached upstream go-sql-driver/mysql,
and with it the whole two-linked-drivers hazard this PR was working
around.

Fixes a startup break found in review. ConnectionDSN injects tls=rds,
and a tls= value is a *name* that only resolves inside the registry of
the driver package that registered it -- Spirit registers "rds" into
block/mysql. Open honoured that; OpenReloadable could not, because the
hot-swap driver embeds upstream and cannot be pointed at the fork. A
MySQL storage pool whose host resolves as RDS failed to open with
"unknown config name: rds", on the startup path, so the server did not
come up. No CI job points storage at an *.rds.amazonaws.com address, so
the break was host-shaped rather than code-shaped and nothing in the
suite could see it.

Mirroring the config into upstream's registry would have worked, but
having one registry is better than keeping two in sync, and dropping the
injection instead would have taken the pool from failing loudly to
connecting in the clear.

pkg/connreload holds the reload machinery, driver-independent: a caller
supplies Resolve (raw DSN -> driver.Connector) and Refused (does this
dial error mean the server rejected these credentials), and the package
owns everything about *when* to reload. That is the subtle part, and it
existed twice -- postgresconn had its own copy, which now goes away.
Both storage pools resolve their DSN through the same secrets
machinery, so a difference in how aggressively they re-resolve it would
have said nothing about either engine.

The reimplementation is not a port. The driver it replaces kept its
reload callback in a package-level variable, so opening a second
reloadable pool silently repointed the first one's reload at the second
one's secret; it had no cooldown, so a secrets-backend outage cost one
resolve per rejected dial; and it pinned the electing dial for the
duration of the reload, holding a pool connection slot. Each of those is
fixed and pinned by a test.

pkg/mysqlerr keeps Number/Is and loses its upstream branch. The helper
is still the right seam -- nothing about a second driver fails to
compile, and the way it breaks is silent -- so a depguard rule now
denies both import paths, with the reason in the message.

Also adds the regression test finding 2 asked for: a reloadable pool
against an RDS host, asserting the injected TLS name resolves *and*
that what the pool dials with verifies the server. Both halves are
needed -- block/mysql applies RDS TLS on its own, so the resolved-trust
assertions alone cannot tell "the name resolved" from "the driver
supplied TLS anyway".

github.com/go-sql-driver/mysql remains in go.mod as an indirect
requirement: testcontainers-go/modules/mysql test-imports it. No
SchemaBot package links it -- verified with go list -deps -test ./...

The sadscan annotations in postgresconn_test.go are incidental: the
scanner reports every finding in a file once the file is touched, and
these are the fake localhost DSNs the suite has always used.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants