DRIVERS-3568 Define PSL support in the Initial DNS Seedlist Discovery Specification

.github/workflows/sync-psl.yml

@@ -0,0 +1,73 @@
+name: Sync Public Suffix List
+
+on:
+  schedule:
+    # 12:00 UTC on the first day of each month.
+    - cron: "0 12 1 * *"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+
+env:
+  PSL: source/public-suffix-list/public_suffix_list.dat
+
+jobs:
+  sync:
+    name: Sync PSL and open a PR
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+
+    steps:
+      - uses: actions/checkout@v4
+      - name: Sync the Public Suffix List
+        run: python3 source/public-suffix-list/etc/sync-psl.py
+      - name: Check whether the list changed
+        id: changed
+        run: |
+          if git diff --quiet -- "$PSL"; then
+            echo "The list is unchanged; nothing to do."
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+      - name: Commit and push
+        id: push
+        if: steps.changed.outputs.changed == 'true'
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          # Each run gets its own branch, so nothing is ever force-pushed over. A run that
+          # collides with an earlier branch from the same day is rejected rather than
+          # overwriting it.
+          today=$(date -u +%Y-%m-%d)
+          branch="sync-psl-$today"
+          git switch -c "$branch"
+          git add -- "$PSL"
+          git commit -m "[$today] Sync the Public Suffix List"
+          git push origin "$branch"
+          echo "branch=$branch" >> "$GITHUB_OUTPUT"
+          echo "today=$today" >> "$GITHUB_OUTPUT"
+      - name: Open a pull request
+        if: steps.changed.outputs.changed == 'true'
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          BRANCH: ${{ steps.push.outputs.branch }}
+          TODAY: ${{ steps.push.outputs.today }}
+        run: |
+          body=$(cat <<'EOF'
+          The upstream [Public Suffix List](https://publicsuffix.org/list/) has changed.
+
+          This pull request was opened automatically by the `sync-psl` workflow, which regenerates
+          `source/public-suffix-list/public_suffix_list.dat` via `source/public-suffix-list/etc/sync-psl.py`.
+
+          Please review the diff before merging. If an earlier sync pull request is still open,
+          merge or close this one and that one together -- they change the same file.
+          EOF
+          )
+          gh pr create \
+            --base master \
+            --head "$BRANCH" \
+            --title "[$TODAY] Sync the Public Suffix List" \
+            --body "$body"

.pre-commit-config.yaml

@@ -89,4 +89,5 @@ repos:
     args: ["-L", "fle,re-use,merchantibility,synching,crate,nin,infinit,te,checkin,aks"]
     exclude: |
             (?x)^(.*\.rst
+            |source/public-suffix-list/public_suffix_list\.dat
             )$

source/index.md

@@ -42,6 +42,7 @@
 - [OpenTelemetry](open-telemetry/open-telemetry.md)
 - [Performance Benchmarking](benchmarking/benchmarking.md)
 - [Polling SRV Records for mongos Discovery](polling-srv-records-for-mongos-discovery/polling-srv-records-for-mongos-discovery.md)
+- [Public Suffix List](public-suffix-list/public-suffix-list.md)
 - [Read and Write Concern](read-write-concern/read-write-concern.md)
 - [Retryable Reads](retryable-reads/retryable-reads.md)
 - [Retryable Writes](retryable-writes/retryable-writes.md)

source/public-suffix-list/etc/sync-psl.py

@@ -0,0 +1,91 @@
+"""Sync the Public Suffix List from publicsuffix.org into this specification.
+
+Usage:
+
+    python source/public-suffix-list/etc/sync-psl.py [--check]
+
+Downloads the upstream list, strips comment and blank lines, and writes the result
+to source/public-suffix-list/public_suffix_list.dat. With --check, does not write
+anything and exits non-zero if the committed file is out of date.
+"""
+
+import argparse
+import sys
+import urllib.request
+from pathlib import Path
+
+PSL_URL = "https://publicsuffix.org/list/public_suffix_list.dat"
+
+# source/public-suffix-list/etc/sync-psl.py -> source/public-suffix-list
+SPEC_DIR = Path(__file__).resolve().parent.parent
+DEST = SPEC_DIR / "public_suffix_list.dat"
+
+
+def fetch():
+    request = urllib.request.Request(PSL_URL, headers={"User-Agent": "mongodb-specifications-sync-psl"})
+    with urllib.request.urlopen(request) as response:
+        data = response.read()
+
+    text = data.decode("utf-8")
+
+    # Sanity check: the upstream file always carries these section markers.
+    for marker in ("// ===END ICANN DOMAINS===", "// ===END PRIVATE DOMAINS==="):
+        if marker not in text:
+            sys.exit(f"Downloaded file is missing expected markers {marker!r}; refusing to write.")
+
+    return text
+
+
+def preprocess(text):
+    """Reduce the upstream list to one rule per line.
+
+    Comment lines (those beginning with "//") and blank lines are both removed, so
+    every line will be a rule.
+    """
+    rules = []
+    for line in text.splitlines():
+        # Upstream rules are not indented, but strip anyway so a stray trailing \r or
+        # space does not end up inside a rule.
+        line = line.strip()
+        if not line or line.startswith("//"):
+            continue
+        rules.append(line)
+
+    if not rules:
+        sys.exit("No rules found after stripping comments; refusing to write.")
+
+    # End the file with exactly one newline.
+    return "\n".join(rules) + "\n"
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--check",
+        action="store_true",
+        help="exit non-zero if the committed list differs from upstream, without writing",
+    )
+    args = parser.parse_args()
+
+    new_text = preprocess(fetch())
+
+    old_text = DEST.read_text(encoding="utf-8") if DEST.exists() else None
+
+    if args.check:
+        if old_text is None:
+            sys.exit(f"{DEST} does not exist; run this script without --check.")
+        if old_text != new_text:
+            sys.exit(f"{DEST} is out of date; run source/public-suffix-list/etc/sync-psl.py.")
+        print(f"{DEST.name} is up to date.")
+        return
+
+    if old_text == new_text:
+        print(f"{DEST.name} is already up to date ({len(new_text.splitlines())} lines).")
+        return
+
+    DEST.write_text(new_text, encoding="utf-8")
+    print(f"Wrote {DEST} ({len(new_text.splitlines())} lines).")
+
+
+if __name__ == "__main__":
+    main()

source/public-suffix-list/public-suffix-list.md

@@ -0,0 +1,184 @@
+# Public Suffix List
+
+- Status: In Progress
+- Minimum Server Version: N/A
+
+______________________________________________________________________
+
+## Abstract
+
+The [Public Suffix List](https://publicsuffix.org/) (PSL) enumerates the DNS suffixes under which the public may
+register names. Determining whether a hostname is a public suffix requires consulting this list; it cannot be derived
+from the hostname alone.
+
+This document vendors the list into this repository as [public_suffix_list.dat](public_suffix_list.dat) and specifies
+how to parse it and determine the public suffix of a domain. Drivers whose language offers a maintained Public Suffix
+List library may use that instead; drivers that do not MUST use the vendored copy, obtained from this repository, so
+that they share a single version-controlled copy of the list.
+
+## META
+
+The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
+"OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).
+
+## Specification
+
+### Terms
+
+#### rule
+
+A single entry in the list, consisting of a sequence of labels separated by dots.
+
+#### public suffix
+
+The portion of a domain under which the public may register names, as determined by the algorithm in
+[Determining the public suffix](#determining-the-public-suffix).
+
+### Obtaining the list
+
+Drivers MUST determine the public suffix of a domain using either of the following:
+
+- An existing Public Suffix List library available in the driver's language.
+- The copy of the list vendored in this repository as [public_suffix_list.dat](public_suffix_list.dat), parsed as
+    described in [Rule syntax](#rule-syntax) and applied as described in
+    [Determining the public suffix](#determining-the-public-suffix).
+
+A driver that does not use a library MUST use the copy vendored here, and MUST obtain it from this repository rather
+than from publicsuffix.org or any other source. How a driver takes delivery of the file is up to it -- copying it into
+its own repository and refreshing that copy from here is expected -- but the contents MUST match
+[public_suffix_list.dat](public_suffix_list.dat) as committed.
+
+Whichever a driver chooses, the parsing/usage of the PSL MUST pass the tests in [tests](tests/).
+
+Drivers SHOULD NOT fetch the list from the network at runtime, and SHOULD instead resolve it from a copy shipped with
+the driver.
+
+A driver using a library is responsible for that library's behavior. In particular, drivers SHOULD confirm that the
+library implements wildcard and exception rules, and that the copy of the list it embeds is kept reasonably current.
+
+#### The vendored file
+
+The vendored file is generated by [etc/sync-psl.py](etc/sync-psl.py) and MUST NOT be edited by hand.
+
+It is encoded as UTF-8 and uses LF line endings, and ends with a single trailing newline. Every line is a single rule:
+comment and blank lines have already been removed, so there are none for a parser to skip.
+
+### Rule syntax
+
+This section and [Determining the public suffix](#determining-the-public-suffix) describe how to parse and apply the
+vendored file. They apply to drivers that use the vendored file; a driver that uses a library satisfies them through
+that library.
+
+A rule takes one of three forms:
+
+| Form           | Example   | Meaning                                                               |
+| -------------- | --------- | --------------------------------------------------------------------- |
+| Ordinary rule  | `com.ac`  | Matches exactly these labels.                                         |
+| Wildcard rule  | `*.ck`    | `*` matches exactly one label -- never zero, and never more than one. |
+| Exception rule | `!www.ck` | Overrides a wildcard rule that would otherwise match.                 |
+
+A `*` only ever appears as the leftmost label of a rule.
+
+All rules are lowercase, and internationalized labels are stored as Unicode rather than Punycode. A driver using the
+vendored file MUST therefore convert one side before comparing, so that a Punycode-encoded hostname (one containing
+`xn--` labels) and the Unicode rules it is compared against are in the same form. Comparing the two forms directly will
+fail to match rules that should match.
+
+### Determining the public suffix
+
+To determine the public suffix of a domain, a driver using the vendored file MUST follow the
+[algorithm published by publicsuffix.org](https://publicsuffix.org/list/):
+
+1. Compare the domain's labels against each rule's labels from right to left, treating `*` as matching any single label.
+    Collect every rule that matches.
+2. If no rule matches, the prevailing rule is `*` -- that is, the rightmost label alone is the public suffix.
+3. If any matching rule is an exception rule, it prevails. Otherwise, the matching rule with the most labels prevails.
+4. If the prevailing rule is an exception rule, remove its leftmost label.
+5. The public suffix is the set of the domain's labels matched by the prevailing rule.
+
+For example, given the rules `ck`, `*.ck`, and `!www.ck`: the public suffix of `a.b.ck` is `b.ck`, because `*.ck`
+prevails; but the public suffix of `www.ck` is `ck`, because the exception rule `!www.ck` prevails and has its leftmost
+label removed.
+
+## Test Plan
+
+The tests in [tests](tests/) verify that a driver parses the list correctly, using the `srvAllowedHostsSuffix`
+connection string option as the vehicle. There is one case per rule form: an ordinary rule, a wildcard rule matched by a
+value that does not appear literally in the list, and an exception rule that overrides a wildcard rule. See
+[tests/README.md](tests/README.md) for the format.
+
+## Design Rationale
+
+### Why the list is vendored
+
+Vendoring the list makes the behavior of a given release reproducible and makes each change to the list an auditable
+commit, and it gives drivers without a suitable language library something to copy rather than each deciding where to
+get the list from.
+
+Runtime fetching is discouraged rather than forbidden. It makes behavior depend on network availability and on when the
+client happened to start, so two clients running the same driver version can disagree about whether a hostname is a
+public suffix. That is a reason to prefer a copy shipped with the driver, but not a reason to block a driver whose
+language ecosystem expects a dependency like this to be resolved at install or start time.
+
+### Why a language library is permitted
+
+Several languages have a well-established PSL library, and requiring drivers in those languages to hand-roll the
+matching algorithm instead would mean reimplementing wildcard and exception handling that is already tested upstream.
+Because the tests assert observable behavior rather than the source of the list, either choice can be verified.
+
+The trade-off is that a library embeds its own copy of the list on its own update schedule, so drivers using one give up
+the guarantee that every driver agrees on the same snapshot. That is acceptable here: the majority of suffixes that
+matter for validating a caller-supplied host pattern are likely to be long-established entries rather than recent
+additions.
+
+### Why comments are stripped
+
+Upstream interleaves rules with comments that carry no normative meaning, including Punycode spellings of the
+internationalized rules. Removing them means every line of the vendored file is a rule, which keeps parsers trivial and
+keeps the diff of a monthly sync limited to actual rule changes.
+
+### Why the ICANN and private sections are not distinguished
+
+Upstream splits the list into an ICANN section, containing suffixes delegated through the DNS registry hierarchy (`com`,
+`co.uk`), and a private section, containing domains whose owners hand out subdomains to unrelated third parties
+(`github.io`, `herokuapp.com`). Because the section markers are comments, they are not preserved, and the two are
+indistinguishable in the vendored file.
+
+This is deliberate: both kinds of suffix are equally unsuitable as a caller-supplied host pattern, which is the use this
+list is vendored for.
+
+## Maintenance
+
+[etc/sync-psl.py](etc/sync-psl.py) regenerates `public_suffix_list.dat` from upstream:
+
+```bash
+python source/public-suffix-list/etc/sync-psl.py
+```
+
+The script downloads the list from <https://publicsuffix.org/list/public_suffix_list.dat>, sanity checks that the
+expected section markers are present, strips comment and blank lines, and writes the result. Passing `--check` reports
+whether the committed file is up to date without writing anything, exiting non-zero if it is stale.
+
+The [sync-psl](../../.github/workflows/sync-psl.yml) GitHub Actions workflow runs the script on the first day of each
+month and opens a pull request when the regenerated file differs from the committed one. It can also be triggered
+manually via `workflow_dispatch`.
+
+## Upstream license and attribution
+
+The Public Suffix List is maintained by the [Mozilla Foundation](https://www.mozilla.org/) and made available under the
+[Mozilla Public License, v. 2.0](https://mozilla.org/MPL/2.0/). Upstream distributes the list with the following header,
+which `etc/sync-psl.py` strips along with the rest of the comments:
+
+```text
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this
+// file, You can obtain one at https://mozilla.org/MPL/2.0/.
+```
+
+That notice applies to [public_suffix_list.dat](public_suffix_list.dat) in this directory. The rules themselves are
+reproduced verbatim and in their original order; only the surrounding comments and blank lines have been removed. The
+notice does not apply to the rest of this repository.
+
+## Changelog
+
+- 2026-08-11: Vendor the Public Suffix List and add a script to sync it.