Skip to content

feat(go-server): add DuckDB security profiles - #1151

Draft
derekperkins wants to merge 39 commits into
mainfrom
codex/duckdb-hardening
Draft

feat(go-server): add DuckDB security profiles#1151
derekperkins wants to merge 39 commits into
mainfrom
codex/duckdb-hardening

Conversation

@derekperkins

@derekperkins derekperkins commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add compat, catalog-only, and local-files security profiles to the Go server binary
  • apply extension, secret, cache, path, external-access, spill, and configuration-lock settings before the query layer starts
  • reject remote primary-database paths and canonicalize explicit local file capabilities
  • regression-test common DuckDB 1.5.5 URI schemes, replacement scans, writes, attachments, symlinks, and concurrent connections
  • document each bundle's guarantees and limitations

Profile contracts

  • compat is the default and preserves existing DuckDB and --load-extensions behavior.
  • catalog-only disables external access outside DuckDB's primary-database internals, disables spill files and dynamic extension behavior, and locks the configuration.
  • local-files adds repeatable --allowed-directory and --allowed-path capabilities to catalog-only. Directory and exact-path grants are read/write capabilities, including COPY and ATTACH.

The strict profiles reject --load-extensions, URI and network-share database paths, and operator-supplied DSN values for profile-owned settings. They remain CLI-local rather than introducing a public cross-package profile API.

DuckDB automatically retains access to its file-backed primary database and related WAL, checkpoint, and recovery paths. Stable symlink escapes are denied, but mutable filesystem state still requires OS or container isolation. Statically linked and core extensions remain available.

The URI regression matrix covers the source-derived http, https, s3, s3a, s3n, gcs, gs, r2, hf, azure, az, and abfss prefixes from the DuckDB 1.5.5 extension-prefix table. This is version-specific best-effort coverage; the native enable_external_access setting is the enforcement boundary.

Commit structure

  1. feat(go-server): add DuckDB security profiles
  2. test(go-server): cover DuckDB security profiles
  3. docs(go-server): document security profiles
  4. fix(go-server): scope profile initialization per connector
  5. docs(go-server): clarify hardening profile tradeoffs
  6. fix(go-server): compose hardening with allowlist defaults

The former standalone startup-exit fix is now inherited from #1150 after the restack, so it was dropped rather than duplicated.

Validation

  • go test -count=1 -tags=duckdb_arrow ./...
  • go test -race -count=1 -tags=duckdb_arrow ./...
  • go test -count=20 -tags=duckdb_arrow . -run '^(TestCatalogOnlyProfileInitializesDistinctConnectors|TestSecurityProfileInitializerIsConcurrent|TestLocalFilesProfileInitializesConcurrentConnections)$'
  • go build -tags=duckdb_arrow ./...
  • go vet -tags=duckdb_arrow ./...
  • golangci-lint run
  • git diff --check origin/codex/function-allowlist...HEAD

Stack

This PR is intentionally separate from and based on #1150, which adds the function allowlist. The database profiles provide the resource boundary that a syntactic function policy cannot infer from string arguments.

Validate every parsed function and operator against an opt-in exact-name policy while preserving the existing blocklist and fail-closed exec behavior.
Preserve an explicitly empty allowlist as deny-all while keeping an omitted flag backward compatible.
Describe exact matching, deny-all semantics, parser-visible helpers, and the version-specific external-resource vectors the syntactic policy cannot resolve.
Verify allowlist denials remain HTTP 403 responses and generic command authorization cannot bypass restricted exec.

@derekperkins derekperkins left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the three commits against codex/function-allowlist, so #1150's contents are out of scope here.

This is in good shape. I spent most of the review trying to get out of catalog-only and local-files and could not:

  • glob, read_text, read_blob, read_csv outside the grant — blocked, including glob('/*') and <allowed>/../<other>/secret.txt traversal.
  • SET, SET SESSION, SET GLOBAL, RESET, and PRAGMA forms of every profile-owned setting — all rejected by lock_configuration.
  • LOAD, INSTALL, FORCE INSTALL — blocked, including LOAD of an extension file planted inside an allowed directory, which was the escalation path I most expected to work.
  • File-backed primary — the automatic grant is the database file and its .wal only; neighbours, directory globs, ATTACH, and COPY into that directory are all blocked.
  • temp_directory='' genuinely prevents spilling rather than silently falling back to a default.

Verified locally on 316ae55 (Go 1.26.5, darwin/arm64, DuckDB 1.5.5): go build, go test -count=1, go test -race -count=1, go vet (all -tags=duckdb_arrow) and golangci-lint run are clean.

Four comments: one code issue that is latent today but silent and total if it is ever hit, the startup exit code, and two documentation points — one of which is asking you to claim a stronger guarantee than the text currently does.

— Claude Opus 5

Comment thread packages/server/duckdb-server-go/security_profiles.go Outdated
Comment thread packages/server/duckdb-server-go/main.go
Comment thread packages/server/duckdb-server-go/README.md Outdated
Comment thread packages/server/duckdb-server-go/README.md
Reject source-qualified function calls while preserving DuckDB's parser-generated main-qualified helper nodes, and report each distinct allowlist violation once.
Document explicit-empty and repeated flag behavior, macro and replacement-scan limits, and the exact DuckDB scheme mapping source.
Route process exit through run so configuration and startup failures signal failure without bypassing deferred connector and database cleanup.
Match written qualifier prefixes without assuming DuckDB preserves the source function token, while avoiding synthesized helpers over qualified columns.
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from 7ab8f93 to 3e7e935 Compare August 13, 2026 16:59
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from 3e7e935 to d4aa901 Compare August 14, 2026 02:03
Function qualifiers do not establish function identity because unqualified calls can resolve through macros and the search path. Match DuckDB's serialized leaf function name like the existing blocklist and document the trusted-catalog requirement.
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from d4aa901 to 6ccaf30 Compare August 14, 2026 02:16
Keep allowlist and blocklist polarity behind named constructors while sharing AST decoding and fail-closed malformed-node handling.
Aggregate normalized AST function names so repeated policy violations produce one deterministic error with an occurrence count while singleton messages remain unchanged.
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from 6ccaf30 to 64c5ea9 Compare August 14, 2026 03:18
Keep the default authority surface pinned to DuckDB 1.5.5 and fail upgrades on unreviewed catalog collisions, side effects, or volatility.
Keep omitted policy unrestricted, while a configured zero value admits reviewed local computation. Exact include and exclude overrides retain a deny-all escape hatch.
Keep the CLI unrestricted when no policy flag is present, and provide explicit include, exclude, and exact-only controls when the allowlist is enabled.
Explain default activation, exact overrides, copy-returning inventories, and the version-pinned boundary that remains outside resource enforcement.
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from 64c5ea9 to a2042eb Compare August 14, 2026 05:04
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from a2042eb to 02a298a Compare August 14, 2026 05:09
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from 02a298a to 5360642 Compare August 14, 2026 05:23
Audit every DuckDB 1.5.5 internal macro so fixed local-compute wrappers are admitted while dynamic dispatch, metadata, identity, time, and delay functions remain excluded.
Keep query policy enforcement separate from the reviewed DuckDB name sets so extension bundles can grow without overwhelming the query package API.
Provide opt-in Spatial and Parquet inventories without changing the reviewed defaults. The helpers authorize names only and deliberately leave resource argument policy to a separate layer.
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch 2 times, most recently from 7cfc602 to de36865 Compare August 14, 2026 06:03
@derekperkins
derekperkins force-pushed the codex/function-allowlist branch from 37fd919 to 1ef7c94 Compare August 14, 2026 06:05
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from de36865 to fde7ccd Compare August 14, 2026 06:06
Represent each DuckDB 1.5.5 core extension with immutable compute, elevated, and complete function inventories so callers can compose policies without expanding query package APIs.
Audit every DuckDB 1.5.5 core extension against its pinned source or runtime and partition callable names into compute and elevated groups. Keep uncertain proprietary MotherDuck names elevated.
Include every reviewed core-extension compute group in the default function policy while keeping resource, stateful, volatile, and source-uncertain names opt-in.
Explain the compute, elevated, and complete inventories for every DuckDB 1.5.5 core extension, including source limitations and the update workflow.
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from fde7ccd to 96185b3 Compare August 14, 2026 07:01
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from 96185b3 to 5aae6db Compare August 14, 2026 11:52
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from 5aae6db to 1da1fb2 Compare August 14, 2026 14:57
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from 1da1fb2 to 6217ca6 Compare August 14, 2026 15:27
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from 6217ca6 to 47e8da3 Compare August 14, 2026 15:52
A full DuckDB 1.5.5 extension pass found the UI URL function missing from the reviewed elevated group. Record the runtime discovery workflow for initial inventories and future upgrades.
Resolve strict profiles before opening the connector and complete DuckDB's one-way configuration lock before constructing the query layer. Keep compatibility behavior as the default.
Exercise compatibility, catalog-only, and local-files behavior against DuckDB. Cover source-derived URI schemes, replacement scans, writes, attachments, symlinks, exact paths, and concurrent physical connections.
Describe the three common configuration bundles, their read and write semantics, and DuckDB's primary-database exception. Record version-specific URI coverage and the remaining filesystem and process-isolation limits.
Create independent one-time initializers for each connector so reusing resolved profile settings cannot leave a separate DuckDB instance unlocked.
Pin DuckDB 1.5.5's exact primary-database sidecar grant and document the repeated-read cost of disabling the external file cache.
@derekperkins
derekperkins force-pushed the codex/duckdb-hardening branch from 47e8da3 to c8abd68 Compare August 14, 2026 16:52
Base automatically changed from codex/function-allowlist to main August 14, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant