feat(go-server): add DuckDB security profiles

README.md

@@ -53,7 +53,7 @@ This repository contains a set of related packages, spanning the core Mosaic arc
 
 * [`duckdb-server`](https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb-server): A Python-based server that runs a local DuckDB instance and support queries over Web Sockets or HTTP, returning data in either [Apache Arrow](https://arrow.apache.org/) or JSON format.
 * [`duckdb-server-rust`](https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb-server-rust): A Rust-based server similar to `duckdb-server` (Python) and `mosaic-duckdb` (Node.js) with additional support for HTTP/2. We are still evaluating what server component works best. DuckDB support for Rust is often delayed compared to Python.
-* [`duckdb-server-go`](https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb-server-go): A Go-based server similar to `duckdb-server` (Python) and `mosaic-duckdb` (Node.js) with additional support for HTTP/2. It has experimental support for multi-tenant access control, function blocklisting, and other features to harden the server for production use.
+* [`duckdb-server-go`](https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb-server-go): A Go-based server similar to `duckdb-server` (Python) and `mosaic-duckdb` (Node.js) with additional support for HTTP/2. It has experimental support for multi-tenant access control, function allowlisting and blocklisting, and other features to harden the server for production use.
 * [`mosaic-duckdb`](https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb): A Promise-based Node.js API to DuckDB, along with a data server that supports transfer of [Apache Arrow](https://arrow.apache.org/) and JSON data over either Web Sockets or HTTP. Due to quality and maintenance issues involving the Node.js DuckDB client and Arrow extension, we recommend using the Python-based `duckdb-server` package instead. However, we retain this package for both backwards compatibility and internal testing use.
 
 ### Examples (`packages/examples`)

packages/server/duckdb-server-go/README.md

@@ -33,7 +33,11 @@ You can customize the server behavior with the following command-line flags:
 -   `--key <path>`: Path to a TLS private key file to enable HTTPS.
 -   `--schema-match-headers`: Comma-separated list of headers to match against schema names for multi-tenant access control (e.g., `X-Tenant-Id,verified-user-id`).
 -   `--load-extensions`: Comma-separated list of extensions to install and load at startup. Use a pipe after the extension name to specify a DuckDB repository alias. Unspecified repositories use DuckDB's default (e.g. `mysql_scanner,netquack|community,aws|core_nightly`).
--   `--function-blocklist`: Comma-separated list of functions to block, useful for blocking functions that may pose security or performance risks. (e.g., 'bigquery_query,read_parquet')`
+-   `--function-blocklist`: Comma-separated list of exact function names to block, useful for blocking functions that may pose security or performance risks (e.g. `bigquery_query,read_parquet`).
+-   `--function-allowlist`: Comma-separated list of exact function names to add to the reviewed defaults. Names are matched case-insensitively, repeated flags accumulate names, and an explicitly empty value enables only the defaults.
+-   `--security-profile`: DuckDB external-resource profile: `compat`, `catalog-only`, or `local-files`. Defaults to `compat`.
+-   `--allowed-directory`: Existing local directory available under `local-files`. Repeat the flag for multiple directories.
+-   `--allowed-path`: Existing local file available under `local-files`. Repeat the flag for multiple files.
 
 By default, the server will look for `localhost.pem` and `localhost-key.pem` in the current directory to enable HTTPS if the `--cert` and `--key` flags are not provided.
 
@@ -79,6 +83,133 @@ errors are logged and returned as sanitized 500 responses. Authorization can all
 and exact SQL, but cannot rewrite SQL or sandbox the shared process, filesystem, network, extensions, catalogs, or
 credentials.
 
+### DuckDB Security Profiles
+
+The installed binary provides three common external-resource configurations:
+
+| Profile | DuckDB resources | Extension behavior | Intended use |
+| --- | --- | --- | --- |
+| `compat` | Preserves DuckDB's current defaults. | Preserves `--load-extensions` and automatic loading. | Trusted local and backwards-compatible deployments. |
+| `catalog-only` | Disables external access outside DuckDB's primary-database internals. | Disables automatic installation and loading; rejects `--load-extensions`. | Queries over an in-memory or local primary catalog. |
+| `local-files` | Adds explicit local files or directories to `catalog-only`. | Same as `catalog-only`. | Catalog queries that also need reviewed local datasets. |
+
+For example:
+
+```sh
+# Preserve the existing behavior.
+duckdb-server-go --security-profile=compat --load-extensions=httpfs
+
+# Use only the primary catalog and a reviewed set of SQL functions.
+duckdb-server-go \
+  --database=/srv/mosaic/catalog.duckdb \
+  --security-profile=catalog-only \
+  --function-allowlist=
+
+# Add one dataset tree and one exact file.
+duckdb-server-go \
+  --security-profile=local-files \
+  --allowed-directory=/srv/mosaic/datasets \
+  --allowed-path=/srv/mosaic/reference.parquet
+```
+
+Both local-file flags are repeatable, and every target must exist at startup. Paths are made absolute, existing symlinks
+are resolved, and URI and network-share paths are rejected. A directory grants DuckDB read and write access throughout
+that tree, including `COPY` and `ATTACH`; an exact path is also a read/write capability, not a read-only grant. Use
+server-owned roots that other processes cannot mutate. DuckDB resolves stable symlink targets, but an in-process setting
+cannot eliminate filesystem races involving later symlink, mount, or path changes; use operating-system or container
+isolation for that boundary.
+
+The two strict profiles apply DuckDB's [security settings](https://duckdb.org/docs/current/operations_manual/securing_duckdb/overview)
+to disable external access and the external file cache, automatic extension installation and loading, community,
+unsigned, and metadata-mismatched extensions, persistent-secret storage, unredacted secret output, and temporary-file
+spilling. They leave no configuration-lock exceptions and lock the resulting settings before the query layer starts.
+Disabling spill files means memory-heavy queries fail instead of writing temporary data. Under `local-files`, repeated
+scans of allowed Parquet files may be slower because DuckDB does not retain their blocks in its in-memory external-file
+cache. Statically linked and core extensions remain available, so these settings are not an extension-free sandbox.
+
+With the bundled DuckDB 1.5.5, the implicit grant for a file-backed primary database consists of the database file and
+the exact sidecar paths `<database>.wal`, `<database>.wal.checkpoint`, and `<database>.wal.recovery`; it does not include
+the containing directory. The `local-files` profile separately adds its configured grants. SQL functions such as
+`read_blob` may therefore read the implicitly granted files when they exist. Combine a strict profile with a function
+allowlist when that distinction matters. The profiles do not add authentication, origin checks, per-user isolation,
+SQL-function policy, or CPU and memory limits, and they do not replace filesystem and network restrictions on the server
+process. They are implemented for the single connector owned by this binary; programs embedding `pkg/query` must
+configure and lock their DuckDB instance before serving requests.
+
+### Function Policies
+
+Use an allowlist when the server should accept only reviewed functions and operators. An explicitly empty value enables
+the defaults without adding application-specific names:
+
+```sh
+duckdb-server-go --function-allowlist=
+```
+
+Without `--function-allowlist`, the server remains unrestricted. The binary intentionally exposes only policy
+activation and exact additions; use a custom binary embedding `pkg/query` for exclusions, exact-only policies, or
+extension groups.
+
+Programs embedding `pkg/query` can apply the same policy and add application functions with:
+
+```go
+query.WithFunctionAllowlist(query.FunctionAllowlistOptions{
+	Include: append(functionset.Spatial.Elevated(), "my_function"),
+})
+```
+
+By default, configured policies use `functionset.DefaultFunctions()`, which contains reviewed built-ins and every
+[core extension](https://duckdb.org/docs/current/core_extensions/overview)'s `Compute()` group. `Elevated()` requires
+explicit admission, and `All()` returns both groups. These Go helpers return fresh slices; the CLI accepts exact names only.
+
+The table records unique names reviewed against DuckDB 1.5.5. A name is elevated if any overload has elevated behavior.
+An empty row means the extension has no reviewed function-call names, not that it has no other capabilities.
+
+| Extension | Compute | Elevated | Classification and status |
+| --- | ---: | ---: | --- |
+| `Autocomplete` | 1 | 3 | Parser check; completion and parser controls are elevated. |
+| `Avro` | 0 | 1 | Reader only. |
+| `AWS` | 0 | 1 | Credential and provider operation. |
+| `Azure` | 0 | 0 | Filesystem integration with no reviewed function-call names. |
+| `Delta` | 2 | 9 | Local parser/test helpers; scans, metadata I/O, and writes are elevated. |
+| `DuckLake` | 1 | 21 | Local hash helper; catalog, scan, metadata, and mutation operations are elevated. |
+| `Encodings` | 0 | 0 | CSV codec integration with no reviewed function-call names. |
+| `Excel` | 2 | 1 | Value conversion; the sheet reader is elevated. |
+| `FTS` | 1 | 2 | Text stemming; index creation and mutation are elevated. |
+| `HTTPFS` | 0 | 0 | Filesystem integration with no reviewed function-call names. |
+| `Iceberg` | 2 | 14 | Value helpers; scans, catalogs, metadata I/O, and writes are elevated. |
+| `ICU` | 179 | 7 | Deterministic collation and calendar computation; current-time names are elevated. |
+| `Inet` | 11 | 0 | IP value operations only. |
+| `JSON` | 33 | 9 | Value parsing and serialization; readers, SQL execution, and plan inspection are elevated. |
+| `Lance` | 0 | 12 | Source-pinned scans and metadata operations. |
+| `MotherDuck` | 0 | 198 | Best-effort observed proprietary runtime snapshot; all names are elevated. |
+| `MySQL` | 0 | 5 | Connector and scanner operations. |
+| `ODBC` | 0 | 11 | Connector and scanner operations. |
+| `Parquet` | 2 | 9 | `VARIANT` conversion; file, metadata, bloom, and key operations are elevated. |
+| `Postgres` | 2 | 8 | Value helpers; connector and scanner operations are elevated. |
+| `Quack` | 3 | 9 | Protocol value helpers; remote and session operations are elevated. |
+| `Spatial` | 151 | 13 | Geometry computation; readers, index/catalog access, random generation, and resource-capable transforms are elevated. |
+| `SQLite` | 0 | 3 | Connector and scanner operations. |
+| `TPCDS` | 2 | 2 | Query and answer text; data generators are elevated. |
+| `TPCH` | 2 | 2 | Query and answer text; data generators are elevated. |
+| `UI` | 0 | 5 | HTTP server lifecycle, URL, and status operations. |
+| `UnityCatalog` | 0 | 4 | Attached-catalog and checkpoint operations; the generated registry is incomplete. |
+| `Vortex` | 0 | 2 | Readers verified against the pinned nested source revision. |
+| `VSS` | 0 | 5 | Index access and management operations. |
+
+These groups authorize names only; extension loading and file or network access are separate concerns. Validation is
+syntactic and name-only: it does not bind function identity, inspect arguments, expand macros or views, recursively inspect
+SQL strings, or cover replacement scans and attached-table binding. Keep catalogs and the search path trusted, and enforce
+resource access outside this policy. Pre-provisioned views and attached tables can deliberately expose curated datasets
+while reader functions remain excluded; catalog integrity and process resource controls then carry the boundary.
+
+In Go, `Exclude` wins over `Include`, and `DisableDefaults` creates an exact-only policy. Omitting
+`WithFunctionAllowlist` is unrestricted; configuring an exact-empty policy denies all function calls. A function
+allowlist cannot be combined with a non-empty blocklist, and any configured function policy rejects `exec` requests.
+
+Spatial compute defaults cover Mosaic rendering over existing geometry data, but the `ST_Read` loader remains elevated.
+Current-time functions are omitted from defaults because persistent cache entries do not expire by default; keyword forms
+such as `CURRENT_DATE` are not function nodes and remain outside this policy.
+
 ### Multi-Tenant Access Control
 
 `schema-match-headers` isn't part of the mosaic server API, but is provided here as an example of how to have
@@ -108,19 +239,19 @@ multiple users / customers share the same DuckDB server instance while restricti
 
 _Note:_ Schema matching authorizes schema references in submitted SQL; it does not isolate the shared DuckDB process,
 filesystem, network, extensions, or credentials. It assumes a single catalog; attached catalogs are outside this policy
-boundary, and explicitly catalog-qualified table, `SHOW`, and function references are rejected. The function blocklist
-applies only to explicit function calls. Schema matching does not restrict catalog metadata returned by functions such as
-`duckdb_tables()` and `pragma_table_info()`. If metadata is sensitive, add the exact metadata-function names exposed by the
-deployment to `--function-blocklist`; wildcard patterns such as `duckdb_*` are not supported, and the list must be reviewed
+boundary, and explicitly catalog-qualified table, `SHOW`, and function references are rejected. Function allowlists and
+blocklists apply only to explicit function calls. Schema matching does not restrict catalog metadata returned by functions
+such as `duckdb_tables()` and `pragma_table_info()`. If metadata is sensitive, allow or block the exact metadata-function
+names exposed by the deployment; wildcard patterns such as `duckdb_*` are not supported, and the policy must be reviewed
 when DuckDB or its extensions change. To restrict file-reading functions, also enable schema matching so DuckDB replacement
 scans such as `FROM 'data.parquet'` are rejected as unqualified table references. These controls are not a sandbox: run the
 server with access only to external resources that are safe for every tenant.
 
-If either `--schema-match-headers` or `--function-blocklist` is configured, `json` and `arrow` requests are limited to
-statements DuckDB can serialize for validation; unsupported forms such as `PRAGMA` and `SET` are rejected, with HTTP
-requests receiving a 400 response. All `exec` requests are also rejected until full-statement authorization is supported.
-This includes every `Coordinator.exec(...)` call, such as data loading, preloading, and DDL/DML. Mosaic pre-aggregation
-also uses `exec` to create schemas and tables, so set `preagg: { enabled: false }` in this mode.
+If `--schema-match-headers`, `--function-blocklist`, or `--function-allowlist` is configured, `json` and `arrow` requests
+are limited to statements DuckDB can serialize for validation; unsupported forms such as `PRAGMA` and `SET` are rejected,
+with HTTP requests receiving a 400 response. All `exec` requests are also rejected until full-statement authorization is
+supported. This includes every `Coordinator.exec(...)` call, such as data loading, preloading, and DDL/DML. Mosaic
+pre-aggregation also uses `exec` to create schemas and tables, so set `preagg: { enabled: false }` in this mode.
 
 ## API
 

packages/server/duckdb-server-go/flags.go

@@ -0,0 +1,33 @@
+package main
+
+import "strings"
+
+type optionalCommaListFlag struct {
+	values []string
+	set    bool
+}
+
+func (f *optionalCommaListFlag) Set(value string) error {
+	f.set = true
+	if value != "" {
+		f.values = append(f.values, strings.Split(value, ",")...)
+	}
+	return nil
+}
+
+func (f *optionalCommaListFlag) String() string {
+	return strings.Join(f.values, ",")
+}
+
+type repeatedStringFlag struct {
+	values []string
+}
+
+func (f *repeatedStringFlag) Set(value string) error {
+	f.values = append(f.values, value)
+	return nil
+}
+
+func (f *repeatedStringFlag) String() string {
+	return strings.Join(f.values, ",")
+}

packages/server/duckdb-server-go/flags_test.go

@@ -0,0 +1,38 @@
+package main
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestOptionalCommaListFlag(t *testing.T) {
+	var flag optionalCommaListFlag
+
+	assert.False(t, flag.set)
+	assert.Empty(t, flag.values)
+
+	require.NoError(t, flag.Set("md5,range"))
+	require.NoError(t, flag.Set("+"))
+	assert.True(t, flag.set)
+	assert.Equal(t, []string{"md5", "range", "+"}, flag.values)
+	assert.Equal(t, "md5,range,+", flag.String())
+}
+
+func TestOptionalCommaListFlagPreservesExplicitEmpty(t *testing.T) {
+	var flag optionalCommaListFlag
+
+	require.NoError(t, flag.Set(""))
+	assert.True(t, flag.set)
+	assert.Empty(t, flag.values)
+	assert.Empty(t, flag.String())
+}
+
+func TestRepeatedStringFlag(t *testing.T) {
+	var flag repeatedStringFlag
+
+	require.NoError(t, flag.Set("/srv/data,2026"))
+	require.NoError(t, flag.Set("/srv/other"))
+	assert.Equal(t, []string{"/srv/data,2026", "/srv/other"}, flag.values)
+}