Six machines, a two-tier PKI, revocation that works, a pipeline that keeps it fed, and an endpoint that reports how much margin is left.
And an application that logs into its database with a password, exactly as it did in Chapter 01.
OT-019, open since Chapter 04: the database still authenticates the application with a
password. Everything built since then has been about certificates, and the one connection that
matters to the business has not used them.
OT-006 stands behind it, open since Chapter 01: the credential lives in the application's
memory for the life of the process.
The two are not the same problem and this chapter closes one of them. Be clear about which, because the difference is the honest half of this chapter and the reason there is a Chapter 13.
Serials, dates and container IDs will differ. This chapter installs packages into running containers, which needs the lab to have network access.
cd "chapters/Chapter 12/lab"
lsExpected: docker-compose.yml, and the directories dev01/, db01/, ca01/, hsm01/,
rootca/ and pub01/.
What this chapter writes is marked ★:
lab/
├── docker-compose.yml Chapter 10
├── dev01/
│ ├── Dockerfile Chapter 01
│ ├── entrypoint.sh Chapter 01
│ ├── initdb.sql Chapter 01
│ ├── fetch-crl.py Chapter 10
│ ├── crl-status.py Chapter 11
│ ├── crontab Chapter 11
│ ├── app/
│ │ ├── config.yaml ★ changed: an identity, and no password anywhere
│ │ └── paymentsvc.py ★ changed: connects with a certificate
│ └── secretstore/
│ ├── secretstore.py Chapter 03
│ ├── secretstore-set.py Chapter 02
│ └── policy.json Chapter 03
├── db01/
│ ├── Dockerfile Chapter 04
│ ├── entrypoint.sh Chapter 04
│ ├── impostor.py Chapter 04
│ └── crontab ★ new: the database becomes a verifier and needs a CRL
├── ca01/
│ ├── Dockerfile Chapter 07
│ ├── entrypoint.sh Chapter 07
│ └── request-cert.sh Chapter 08
├── hsm01/
│ ├── Dockerfile Chapter 07
│ ├── entrypoint.sh Chapter 07
│ ├── hsm-init.sh Chapter 07
│ ├── ica-init.sh Chapter 08
│ ├── sign-leaf.sh Chapter 10
│ ├── signd.py Chapter 10
│ ├── stop-signd.sh Chapter 08
│ ├── policy.json ★ changed: a workload name joins the policy
│ ├── ca.cnf Chapter 09
│ ├── crl-refresh.sh Chapter 09
│ ├── revoke-cert.sh Chapter 09
│ └── crontab Chapter 11
├── rootca/ Chapter 09, unchanged
└── pub01/ Chapter 11, unchanged
sudo docker start db01 ca01 hsm01 dev01 pub01
sudo docker exec -d -u signd hsm01 \
sh -c 'python3 /usr/local/bin/signd >>/var/log/signd.out 2>&1'
sudo docker exec -d -u pub pub01 sh -c 'python3 /usr/local/bin/pubd >>/var/log/pubd.out 2>&1'
for h in hsm01 pub01 dev01; do sudo docker exec -d $h cron; done
sleep 2
sudo docker exec dev01 sh -c '
for i in $(seq 1 30); do pg_isready -q -h 127.0.0.1 -p 5432 && break; sleep 1; done
pg_ctlcluster 15 main stop'
sudo docker exec -d -u secretstore dev01 \
sh -c 'python3 /opt/secretstore/secretstore.py >>/var/log/secretstore.out 2>&1'
sleep 1
sudo docker exec -d -u paymentsvc dev01 \
sh -c 'python3 /opt/paymentsvc/paymentsvc.py >>/var/log/paymentsvc.out 2>&1'
sleep 2
curl -s http://127.0.0.1:8080/payments/1001/status
curl -s http://127.0.0.1:8080/healthz; echoExpected: the payment record, and {"status": "ok", ...} with both lists reported ok.
The application authenticates like this today:
curl -s http://127.0.0.1:8080/credinfo; echoExpected: "db_user": "paymentsvc_a" or _b, a credential_version, and "sslmode": "verify-full".
That is Chapter 02's design and it works. Two login roles, one live at a time, rotated by
PROC-01 in six steps with no outage, fetched from SVC-02 over a Unix socket by a process the
kernel identifies. Nothing about it is careless.
And it has one property nothing above can fix: the database knows the secret.
sudo docker exec db01 su postgres -c \
"psql -tAc \"SELECT rolname, substring(rolpassword,1,14) FROM pg_authid \
WHERE rolname LIKE 'paymentsvc%'\""Expected: the two login roles and the beginning of a SCRAM-SHA-256$ verifier for each.
That is not the password, and SCRAM is a real improvement over storing one: what is on disk is a salted, iterated verifier, and Chapter 01 §5 measured that the password itself never crosses the wire. But the shape of the thing has not changed. Two parties have to agree on a value. The application holds it, the database holds something derived from it, and a compromise of either end is a compromise of the credential.
Everything PROC-01 does is an attempt to make that survivable: rotate often, keep two roles so
there is no window, verify convergence before retiring the old one. Six steps, two roles, one
store, one policy, all of it in service of a shared value.
The alternative is not a better secret. It is not sharing one. A private key is held by exactly one party. The database can verify a signature made with it and has never seen it, cannot leak it, and cannot be made to give it up under any amount of compromise. That is a property no rotation schedule buys.
We have had the machinery to do this since Chapter 05.
There is a shift here that is easy to miss because the mechanism is familiar.
Every certificate this estate has issued names a machine: db01.lab.simurgh.example,
hsm01.lab.simurgh.example, ca01.lab.simurgh.example. The name is a hostname, the SAN matches
what a client dials, and the identity being asserted is "I am this host".
APP-01 is not a host. It is a process running as ACC-03 on HOST-01, alongside SVC-02 and,
in Chapter 03, a deliberately similar reportsvc account that exists to prove the store can tell
them apart. A certificate naming dev01.lab.simurgh.example would authenticate the machine,
and every process on that machine that can read the key would inherit it.
So the certificate is named for the workload: paymentsvc.
Figure 12.1 — what the certificate names
flowchart TB
subgraph HOSTID["Host identity: what Chapters 05 to 11 issued"]
direction TB
H1["CERT for dev01.lab.simurgh.example"]:::app
P1["APP-01 paymentsvc"]:::app
P2["SVC-02 secretstore"]:::app
P3["ACC-07 reportsvc"]:::app
H1 -.->|"any process that can<br/>read the key is dev01"| P1
H1 -.-> P2
H1 -.-> P3
end
subgraph WORKID["Workload identity: what Chapter 12 issues"]
direction TB
W1["★ CERT-11 for paymentsvc"]:::best
Q1["APP-01 paymentsvc<br/>key mode 0400, owner ACC-03"]:::app
Q2["SVC-02 secretstore"]:::retired
Q3["ACC-07 reportsvc"]:::retired
W1 ==>|"one process,<br/>and the file mode says which"| Q1
W1 -.->|"cannot read the key"| Q2
W1 -.->|"cannot read the key"| Q3
end
classDef app fill:#e5e7eb,stroke:#6b7280,stroke-width:1px,color:#111827
classDef best fill:#dcfce7,stroke:#15803d,stroke-width:3px,color:#052e16
classDef retired fill:#f8fafc,stroke:#cbd5e1,stroke-width:1px,stroke-dasharray:2 4,color:#94a3b8
style HOSTID fill:#f8fafc,stroke:#475569,stroke-width:1px
style WORKID fill:#f8fafc,stroke:#475569,stroke-width:1px
Read the dotted lines in the upper box. A host certificate is a credential every process on
that host shares, limited only by file permissions, which is exactly the argument Chapter 03 made
about SEC-01 before SO_PEERCRED existed.
And read what the lower box depends on, because the honesty matters: the separation is a file
mode. 0400 owned by ACC-03 is what stops SVC-02 from becoming paymentsvc, and root on
HOST-01 still reads everything. That is OT-004, and Chapter 12 does not touch it.
This is also the shift Kubernetes will force. A pod has no stable hostname worth naming and may run anywhere; the thing worth authenticating is the workload. Getting there now, with certificates we understand, means Stage 4 is a change of mechanism rather than a change of idea.
The key is generated where it will be used and never travels. D-036 and D-044, unchanged
since Chapter 05.
It needs somewhere to live, and it is not beside the configuration:
sudo docker exec dev01 sh -c '
mkdir -p /var/lib/paymentsvc
chown paymentsvc:paymentsvc /var/lib/paymentsvc
chmod 0700 /var/lib/paymentsvc
ls -ld /opt/paymentsvc /var/lib/paymentsvc'Expected: /opt/paymentsvc owned by root, and /var/lib/paymentsvc owned by paymentsvc with
mode 0700.
Chapter 10 learned this the hard way and it applies again. /opt/paymentsvc is root-owned so
that APP-01 cannot rewrite its own configuration, which has been true since Chapter 01.
Creating a key there needs write permission on the directory, not merely on a file, and so
does replacing it during an incident, which §7 does. An unprivileged workload that owns an
identity needs a directory of its own, and the reflex fix of widening /opt/paymentsvc would
undo a control put there eleven chapters ago.
sudo docker exec -u paymentsvc dev01 sh -c '
openssl ecparam -name prime256v1 -genkey -noout -out /var/lib/paymentsvc/client.key
chmod 0400 /var/lib/paymentsvc/client.key
openssl req -new -key /var/lib/paymentsvc/client.key \
-out /tmp/paymentsvc.csr -subj "/CN=paymentsvc"
ls -l /var/lib/paymentsvc/client.key'Expected: -r-------- 1 paymentsvc paymentsvc ... client.key.
The CSR goes to /tmp because it is public and about to travel.
The authority will not issue this yet. POL-02 decides who may ask for which name, and nobody
has ever asked for paymentsvc:
sudo docker cp dev01:/tmp/paymentsvc.csr /tmp/paymentsvc.csr
sudo docker cp /tmp/paymentsvc.csr ca01:/opt/ca-client/requests/paymentsvc.csr
sudo docker exec ca01 chown ca:ca /opt/ca-client/requests/paymentsvc.csr
sudo docker exec -u ca ca01 request-cert /opt/ca-client/requests/paymentsvc.csr paymentsvcExpected:
request-cert: refused or unreachable:
{"error": "denied", "you_are": "ca01.lab.simurgh.example", "requested": "paymentsvc",
"detail": "POL-02 does not permit this caller to request this name"}
That refusal is the system working, and it is worth pausing on. The request was authentic,
the caller was who it claimed to be, and the name was refused because nobody had decided that
ca01 may speak for a workload. Chapter 07 built that gate and this is the first time it has
said no to a legitimate request.
Add the rule:
{
"ca01.lab.simurgh.example": [
"db01.lab.simurgh.example",
"paymentsvc"
]
}sudo docker cp hsm01/policy.json hsm01:/etc/signd/policy.json
sudo docker exec hsm01 chmod 0644 /etc/signd/policy.json
sudo docker exec -u ca ca01 request-cert /opt/ca-client/requests/paymentsvc.csr paymentsvc
sudo docker exec -u ca ca01 \
openssl x509 -in /opt/ca-client/issued/paymentsvc.crt -noout -ext extendedKeyUsageExpected: an issued certificate, and then:
X509v3 Extended Key Usage:
TLS Web Server Authentication
POL-02 is re-read on every request, which Chapter 07 chose deliberately, so no restart was
needed. Note also what the policy now contains: a hostname and a workload name, side by side,
with nothing in the file marking which is which. That is OT-016's complaint arriving at
POL-02.
serverAuth, on a certificate whose entire job is to authenticate a client. PostgreSQL
refuses that with sslv3 alert unsupported certificate, which is the error Chapter 07 §5.1
spends a page on: it names neither the field nor the purpose, and it arrives before any policy is
consulted.
sign-leaf has had a --client flag since Chapter 07. SVC-03 never passed it:
sudo docker exec -u signd hsm01 grep -n "sign-leaf" /usr/local/bin/signd | head -3Expected, before the fix below, a call with no --client anywhere in it.
Nobody noticed for five chapters, and the reason is worth sitting with. The only client
certificate this estate had was CERT-07, issued by hand on hsm01 in Chapter 07 §5,
because of the bootstrap problem: signd could not issue the certificate signd's own caller
needed. The flag was written, exercised by hand, and never reachable through the API.
A capability that exists in a tool and not in the interface to it is a capability nobody has.
Wire it through. signd takes a usage field and request-cert gains the matching flag:
sudo docker cp hsm01/signd.py hsm01:/usr/local/bin/signd
sudo docker cp ca01/request-cert.sh ca01:/usr/local/bin/request-cert
sudo docker exec hsm01 chmod 0755 /usr/local/bin/signd
sudo docker exec ca01 chmod 0755 /usr/local/bin/request-cert
sudo docker exec -u signd hsm01 stop-signd
sudo docker exec -d -u signd hsm01 \
sh -c 'python3 /usr/local/bin/signd >>/var/log/signd.out 2>&1'
sleep 1
sudo docker exec -u ca ca01 request-cert --client \
/opt/ca-client/requests/paymentsvc.csr paymentsvc
sudo docker exec -u ca ca01 openssl x509 -in /opt/ca-client/issued/paymentsvc.crt \
-noout -subject -ext extendedKeyUsageExpected: usage: client, an issued certificate, subject=CN = paymentsvc, and:
X509v3 Extended Key Usage:
TLS Web Client Authentication
POL-02 governs names and says nothing about usages. A caller permitted to request a name
may now request it as either kind of certificate. That is not obviously wrong, since the name is
what the certificate asserts, and it is unexamined, which is the complaint. OT-042.
Install it:
The two files that changed, in full:
#!/usr/bin/env python3
"""SVC-03 signd, the signing service on HOST-04 hsm01.
The token lives here and nothing else does. Callers send a certificate request
over mTLS and get a certificate back; the key never leaves this machine, and
after Chapter 07 it never leaves this machine's token either.
CHAPTER 08 CHANGES ALMOST NOTHING HERE, WHICH IS THE POINT. The key this
service signs with is now an intermediate rather than a root, and the code did
not need to know: sign-leaf changed a token label and this file gained one
field in its reply. What changed is what a compromise of this host costs. An
attacker who takes this machine can issue certificates under CERT-09 until
CERT-09 is replaced, and replacing it is a ceremony on a machine that is
switched off, which touches no client at all. Before Chapter 08 the same
attacker took the estate's trust anchor.
The one field is `chain`. A leaf signed by an intermediate does not verify
against the root on its own, so every reply now carries the issuer beside the
certificate. Callers that ignore it get a certificate that works nowhere, with
an error naming neither this service nor the missing file.
Three things this deliberately does, and one it deliberately does not.
DOES take the caller's identity from the TLS layer rather than from the request.
The client certificate is checked against CERT-05 by the kernel of the TLS
handshake before a single byte of the request body is read, and the name comes
out of that certificate. It is Chapter 03's SO_PEERCRED argument moved onto a
network: an observation, not a claim.
DOES consult POL-02 on every request. mTLS answers "did our authority sign your
certificate" and nothing else. Any holder of any certificate we ever issued can
open this connection, which is exactly what Chapter 07 section 5 demonstrates,
so authentication and authorization stay separate questions.
DOES record every decision, allowed or refused, with the identity the TLS layer
reported and the name that was requested. SVC-02 has done this since Chapter 02
and the authority has never done it at all.
DOES NOT hold the key. It shells out to sign-leaf, which asks the token. This
process could be compromised entirely and the attacker would gain the ability to
request signatures while this process lives, not a key they can keep.
"""
import http.server
import json
import os
import re
import ssl
import subprocess
import sys
import threading
import tempfile
import datetime
LISTEN = ("0.0.0.0", 8443)
CA_CRT = "/var/lib/ca/ca.crt" # CERT-08, the root we verify clients
# against. Holds TWO roots during
# the Chapter 08 overlap, because a
# trust anchor is a bundle and not
# a certificate. That is the fact
# that made all three of this
# build's root migrations survivable.
SRV_CRT = "/var/lib/ca/signd.crt" # CERT-06 followed by CERT-09. This
# file must hold the CHAIN, not the
# leaf alone, or no client can build
# a path from us to the root.
SRV_KEY = "/var/lib/ca/signd.key"
ICA_CRT = "/var/lib/ca/ica.crt" # CERT-09, returned with every leaf
CRL = "/var/lib/ca/crl.pem" # CRL-01, both lists, public
PUBLIC_LISTEN = ("0.0.0.0", 8080) # see PublicHandler
POLICY = "/etc/signd/policy.json" # POL-02
AUDIT = "/var/log/signd-audit.log"
# A name we will sign for has to look like a hostname. This is not a security
# control, it is a guard against a malformed request becoming a malformed
# openssl invocation; POL-02 is the control.
NAME_RE = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$")
def audit(caller, requested, decision, detail=""):
"""Append one line per decision. Allowed and refused both, or the log only
tells you about the requests that worked."""
line = "\t".join([
datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
f"caller={caller}", f"requested={requested}",
f"decision={decision}", detail,
])
with open(AUDIT, "a") as fh:
fh.write(line + "\n")
print(line, flush=True)
def load_policy():
"""Re-read on every request, so an edit takes effect on the next call.
Inefficient and deliberate, exactly as POL-01 is in SVC-02."""
with open(POLICY) as fh:
return json.load(fh)
class Handler(http.server.BaseHTTPRequestHandler):
server_version = "signd/1.0"
def _json(self, code, obj):
body = json.dumps(obj).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def peer_name(self):
"""Who is calling, according to the certificate they proved they hold.
getpeercert() returns the parsed client certificate. It is present only
because the context was built with CERT_REQUIRED, so by the time this
runs the chain has already been verified against CERT-05. Nothing here
trusts anything the caller wrote in the request.
"""
cert = self.connection.getpeercert()
if not cert:
return None
for dns in [v for k, v in cert.get("subjectAltName", ()) if k == "DNS"]:
return dns
subject = dict(x[0] for x in cert.get("subject", ()))
return subject.get("commonName")
def do_POST(self):
caller = self.peer_name() or "unknown"
if self.path != "/v1/sign":
return self._json(404, {"error": "not found"})
length = int(self.headers.get("Content-Length") or 0)
if length <= 0 or length > 16384:
audit(caller, "-", "deny", "detail=bad request size")
return self._json(400, {"error": "csr required, 16k max"})
try:
req = json.loads(self.rfile.read(length))
csr, fqdn = req["csr"], req["fqdn"]
extra = req.get("alt_names", [])
# CHAPTER 12. sign-leaf has had --client since Chapter 07 and this
# service never passed it, because the only client certificate in
# the estate was issued by hand during the bootstrap in Chapter 07
# section 5. The first one requested through the API arrived five
# chapters later, stamped serverAuth, and was refused by PostgreSQL
# with `sslv3 alert unsupported certificate`: the same error
# Chapter 07 section 5.1 spends a page on, and the same cause.
#
# A capability that exists in a tool and not in the interface to it
# is a capability nobody has.
usage = req.get("usage", "server")
except Exception:
audit(caller, "-", "deny", "detail=malformed json")
return self._json(400, {"error": "malformed request"})
if usage not in ("server", "client"):
audit(caller, fqdn, "deny", f"detail=unknown usage {usage!r}")
return self._json(400, {"error": f"usage must be server or client: {usage}"})
for name in [fqdn] + list(extra):
if not isinstance(name, str) or not NAME_RE.match(name):
audit(caller, fqdn, "deny", f"detail=bad name {name!r}")
return self._json(400, {"error": f"not a hostname: {name}"})
# POL-02. mTLS said who is calling. This says whether they may speak
# for the name they are asking for, which is a different question and
# the one Chapter 03 section 7.4 is about.
# POL-02 answers which NAMES this caller may request. It has nothing
# to say about which USAGE, so a caller permitted to request a name
# may request it as either a server or a client certificate. That is
# OT-042 and it is not obviously wrong, because the name is what the
# certificate asserts; it is unexamined, which is the complaint.
allowed = load_policy().get(caller, [])
if fqdn not in allowed:
audit(caller, fqdn, "deny", "detail=POL-02 does not permit")
return self._json(403, {
"error": "denied", "you_are": caller, "requested": fqdn,
"detail": "POL-02 does not permit this caller to request this name",
})
with tempfile.TemporaryDirectory() as tmp:
path = os.path.join(tmp, "req.csr")
with open(path, "w") as fh:
fh.write(csr)
# sign-leaf owns the token interaction. This process never sees a
# key, and could not leak one if it were compromised.
argv = ["/usr/local/bin/sign-leaf"]
if usage == "client":
argv.append("--client")
argv += [path, fqdn] + list(extra)
proc = subprocess.run(argv, capture_output=True, text=True)
if proc.returncode != 0:
audit(caller, fqdn, "error", f"detail=sign-leaf exit {proc.returncode}")
return self._json(500, {"error": "signing failed"})
issued = f"/var/lib/ca/issued/{fqdn}.crt"
with open(issued) as fh:
cert = fh.read()
# The issuer travels with the certificate. It is public, it is the
# same bytes for every caller, and sending it is the difference
# between a certificate that works and one that fails at whichever
# client is unlucky enough to be first.
with open(ICA_CRT) as fh:
chain = fh.read()
audit(caller, fqdn, "allow", f"detail=issued {fqdn} usage={usage}")
return self._json(200, {"certificate": cert, "chain": chain,
"issued_for": fqdn})
def do_GET(self):
if self.path == "/healthz":
return self._json(200, {"status": "ok"})
return self._json(404, {"error": "not found"})
def log_message(self, *args):
"""Silence the default access log. Everything that matters goes through
audit(), which records the verified identity rather than an IP."""
return
class PublicHandler(http.server.BaseHTTPRequestHandler):
"""Serves the artefacts that are public by construction, over plain HTTP.
WHY THIS IS NOT ON THE mTLS LISTENER. A client needs CERT-08 and CRL-01 in
order to verify anybody, including us. Requiring a verified connection to
collect the things you need in order to verify is a bootstrap that does not
close. So these are served unauthenticated.
WHY PLAIN HTTP IS NOT A MISTAKE. Every file here is signed by a key the
client already trusts, carries a serial or a crlNumber, and has dates in
it. A forger gains nothing from controlling the channel because they cannot
produce a signature. Verifying the transport would be a second and weaker
control than verifying the content, and it would tempt a client into
skipping the check that actually matters.
WHAT IT IS NOT SAFE AGAINST is replay: an authentic OLD file, served by
anybody. That is not fixable here, at the source, because the file is
genuine. It is fixed at the client, which remembers the highest crlNumber
it has installed and refuses to go backwards.
THE DEVIATION, STATED. D-054 says hsm01 carries nothing a general purpose
host carries, and this is a second listening socket on the machine that
holds the key. A real estate has the CA push its artefacts outward and run
no inbound listener at all. This one cannot: adding a shared volume to this
service would make compose recreate the container, and recreating it
destroys ica-token and everything Chapter 08 and 09 built inside it. The
surface is one GET over an allow-list of two filenames, and pub01 exists so
that nothing except pub01 ever needs to reach it.
"""
server_version = "signd-public/1.0"
# An allow-list, not a directory. Serving a path the caller supplies is how
# a static file server becomes a way to read /var/lib/ca/ica-pin.
FILES = {
"/crl.pem": (CRL, "application/x-pem-file"),
"/ca-bundle.pem": (None, "application/x-pem-file"), # assembled below
}
def _send(self, code, body, ctype="text/plain"):
if isinstance(body, str):
body = body.encode()
self.send_response(code)
self.send_header("Content-Type", ctype)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_GET(self):
if self.path not in self.FILES:
return self._send(404, "not found\n")
path, ctype = self.FILES[self.path]
try:
if self.path == "/ca-bundle.pem":
# Assembled on every request rather than cached, so it cannot
# be stale. Two certificates: the anchor, and the intermediate
# a client needs in order to check the intermediate's own CRL.
with open(CA_CRT) as fh:
body = fh.read()
with open(ICA_CRT) as fh:
body += fh.read()
else:
with open(path) as fh:
body = fh.read()
except OSError as exc:
audit("public", self.path, "error", f"detail={exc}")
return self._send(503, "not available yet\n")
return self._send(200, body, ctype)
def do_POST(self):
return self._send(405, "read only\n")
do_PUT = do_DELETE = do_POST
def log_message(self, *args):
return
def main():
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_cert_chain(SRV_CRT, SRV_KEY)
ctx.load_verify_locations(CA_CRT)
# Without this line the service would accept anyone and read a name out of
# nothing. With it, the handshake fails before do_POST is ever entered.
ctx.verify_mode = ssl.CERT_REQUIRED
ctx.minimum_version = ssl.TLSVersion.TLSv1_2
# The public listener runs in a daemon thread beside the mTLS one. It is
# started first so that a client polling for the CRL gets an answer even
# while the signing side is still coming up.
pub = http.server.ThreadingHTTPServer(PUBLIC_LISTEN, PublicHandler)
threading.Thread(target=pub.serve_forever, daemon=True).start()
print(f"signd public artefacts on {PUBLIC_LISTEN[0]}:{PUBLIC_LISTEN[1]}, "
"no authentication, read only", flush=True)
srv = http.server.ThreadingHTTPServer(LISTEN, Handler)
srv.socket = ctx.wrap_socket(srv.socket, server_side=True)
print(f"signd listening on {LISTEN[0]}:{LISTEN[1]}, mTLS required", flush=True)
srv.serve_forever()
if __name__ == "__main__":
sys.exit(main())#!/bin/sh
# PROC-02, rewritten. Requests a certificate from SVC-03 on hsm01.
#
# request-cert <csr-file> <fqdn> [additional-dns-name ...]
#
# Chapter 06's version of this script signed. Chapter 07's asked. Everything
# that touches the signing key happens on a machine this one cannot log in
# to, so what is left here is a client: build a request, present a
# certificate proving who we are, and receive a certificate or a refusal.
#
# Note what ca01 does not have, because it is the point of Chapter 07. No
# token, no PIN, no softhsm group, no signing key of any kind. Root here can
# read CERT-07 and its key, which is enough to ASK, and POL-02 decides what
# asking gets you. It is not enough to sign anything.
#
# CHAPTER 08 STOPS PRINTING THE CERTIFICATE TO STDOUT, and the reason is the
# hierarchy rather than taste. There used to be one file to install. There
# are now two, the leaf and the chain, and a script that prints one of them
# to a pipe is a script whose caller loses the other one without being told.
# So this writes both, next to each other, and prints where they went.
#
# Install the CHAIN, not the leaf. Every failure this chapter demonstrates
# comes from someone installing the leaf on its own.
set -eu
SIGND=https://hsm01.lab.simurgh.example:8443/v1/sign
DIR=/opt/ca-client
ANCHOR=$DIR/ca.crt # CERT-08, so we verify the service
CLIENT_CRT=$DIR/ca01.crt # CERT-07 followed by CERT-09: our own chain.
# A client presents a chain for the same
# reason a server does, and gets the same
# unhelpful error when it does not.
CLIENT_KEY=$DIR/ca01.key
ISSUED=$DIR/issued
# Chapter 12 adds --client, mirroring the flag sign-leaf has had since
# Chapter 07. It was never reachable from here, so every certificate this
# API has ever issued has been a server certificate, and the first client
# one requested through it was refused by PostgreSQL with `sslv3 alert
# unsupported certificate`.
USAGE=server
if [ "${1:-}" = "--client" ]; then
USAGE=client
shift
fi
if [ $# -lt 2 ]; then
echo "usage: request-cert [--client] <csr-file> <fqdn> [additional-dns-name ...]" >&2
exit 2
fi
CSR="$1"; FQDN="$2"; shift 2
[ -r "$CSR" ] || { echo "request-cert: cannot read CSR: $CSR" >&2; exit 1; }
[ -r "$CLIENT_KEY" ] || { echo "request-cert: cannot read CERT-07's key. Run as 'ca'." >&2; exit 1; }
# Build the JSON body without a JSON library, because this host has no
# python3 and this script should not be the reason it acquires one. The CSR
# is PEM, so the only escaping needed is the newlines.
ALT=""
for n in "$@"; do ALT="$ALT\"$n\","; done
ALT=$(printf '%s' "$ALT" | sed 's/,$//')
BODY=$(printf '{"csr": "%s", "fqdn": "%s", "alt_names": [%s], "usage": "%s"}' \
"$(sed ':a;N;$!ba;s/\n/\\n/g' "$CSR")" "$FQDN" "$ALT" "$USAGE")
# --cacert is not optional. Without it this client would hand a request to
# anything answering on that name, which is Chapter 04 section 7 in reverse:
# there, the app trusted an impostor database; here, we would trust an
# impostor authority and take back a certificate it signed.
RESP=$(curl -sS --fail-with-body \
--cacert "$ANCHOR" --cert "$CLIENT_CRT" --key "$CLIENT_KEY" \
-H 'Content-Type: application/json' \
-X POST "$SIGND" -d "$BODY") || {
echo "request-cert: refused or unreachable:" >&2
echo "$RESP" >&2
exit 1
}
# Pull the two PEMs out of the JSON reply. sed rather than a parser, for the
# same reason as above; the fields are ours and their order is fixed.
mkdir -p "$ISSUED"
printf '%s' "$RESP" | sed -n 's/.*"certificate": "\(.*\)", "chain".*/\1/p' \
| sed 's/\\n/\n/g' > "$ISSUED/$FQDN.crt"
printf '%s' "$RESP" | sed -n 's/.*"chain": "\(.*\)", "issued_for".*/\1/p' \
| sed 's/\\n/\n/g' > "$ISSUED/ica.crt"
cat "$ISSUED/$FQDN.crt" "$ISSUED/ica.crt" > "$ISSUED/$FQDN.chain.crt"
chmod 0644 "$ISSUED/$FQDN.crt" "$ISSUED/ica.crt" "$ISSUED/$FQDN.chain.crt"
echo "usage: $USAGE"
echo "leaf: $ISSUED/$FQDN.crt"
echo "chain: $ISSUED/$FQDN.chain.crt <- install this one"
openssl x509 -in "$ISSUED/$FQDN.crt" -noout -subject -issuer -datessudo docker cp ca01:/opt/ca-client/issued/paymentsvc.chain.crt /tmp/client.crt
sudo docker cp /tmp/client.crt dev01:/var/lib/paymentsvc/client.crt
sudo docker exec dev01 sh -c '
chown paymentsvc:paymentsvc /var/lib/paymentsvc/client.crt
chmod 0444 /var/lib/paymentsvc/client.crt
openssl x509 -in /var/lib/paymentsvc/client.crt -noout -subject -ext extendedKeyUsage
grep -c "BEGIN CERTIFICATE" /var/lib/paymentsvc/client.crt'Expected: subject=CN = paymentsvc, TLS Web Client Authentication, and 2.
Two certificates in the file, because sign-leaf writes the chain and D-064 says the chain
travels with the certificate. §5 explains why that matters less here than it did in Chapter 10,
and why we do it anyway.
db01 has never checked a client certificate. Giving it that job means giving it three things it
does not have: the anchors, a revocation list, and a way to keep the list fresh.
sudo docker exec db01 sh -c 'apt-get update -qq >/dev/null 2>&1 && \
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \
python3 cron >/dev/null 2>&1; command -v python3 cron'Expected: /usr/bin/python3 and /usr/sbin/cron.
That is the third chapter in a row to install something into a running container. Chapter 10
added a listener to hsm01, Chapter 11 added cron to three machines, and this adds python3
and cron to a fourth. Every one of them was forced by the same fact: hsm01 cannot be rebuilt
without destroying KEY-06, so nothing is ever rebuilt, so the Dockerfiles describe a system
that has not existed for three chapters.
OT-020 has stopped being a tidiness complaint. A reconciliation pass follows this chapter,
and it will bring every Dockerfile up to date with what its machine contains without rebuilding
anything, because the substrate definition may reset freely and the system state may not.
sudo docker exec db01 sh -c '
mkdir -p /var/lib/postgresql/crl
chown postgres:postgres /var/lib/postgresql/crl
curl -sS -o /etc/postgresql/15/main/ca-bundle.pem \
http://pub01.lab.simurgh.example/ca-bundle.pem
chown postgres:postgres /etc/postgresql/15/main/ca-bundle.pem
chmod 0644 /etc/postgresql/15/main/ca-bundle.pem
grep -c "BEGIN CERTIFICATE" /etc/postgresql/15/main/ca-bundle.pem'Expected: 2, the root and the intermediate.
Deploy the agent and put it on a clock:
sudo docker cp dev01/fetch-crl.py db01:/usr/local/bin/fetch-crl
sudo docker cp dev01/crl-status.py db01:/usr/local/bin/crl-status
sudo docker cp db01/crontab db01:/var/lib/postgresql/crontab
sudo docker exec db01 sh -c '
chmod 0755 /usr/local/bin/fetch-crl /usr/local/bin/crl-status
chown postgres:postgres /var/lib/postgresql/crontab
crontab -u postgres /var/lib/postgresql/crontab'
sudo docker exec -d db01 cron
sudo docker exec -u postgres db01 fetch-crl \
--url http://pub01.lab.simurgh.example/crl.pem \
--anchors /etc/postgresql/15/main/ca-bundle.pem \
--install /var/lib/postgresql/crl/crl.pem \
--state /var/lib/postgresql/crl/state.jsonExpected: installed: /var/lib/postgresql/crl/crl.pem, with both authorities and their numbers.
# The database, on a timer. Installed for `postgres` on HOST-02.
#
# crontab -u postgres /var/lib/postgresql/crontab
#
# WHY THE DATABASE NOW HAS A CRON JOB AT ALL. From Chapter 12 db01 verifies
# client certificates, which means it checks revocation, which means it needs
# a current CRL. It has become the estate's second verifier, and a verifier
# that cannot get a fresh list is a verifier that refuses everybody.
#
# That is the half of OT-037 this chapter closes and the new risk it brings.
# Before Chapter 12, a stale CRL on db01 was impossible because db01 had no
# CRL. Now a stale one takes the database offline for every client at once,
# which is a larger blast radius than the same failure on dev01.
#
# EVERY THIRTY MINUTES, matching dev01. Both are clients of the same
# publication point with the same seven day list, and there is no reason for
# them to disagree about how far behind they are willing to be.
#
# WHAT IS MISSING, and it is deliberate rather than forgotten: nothing on
# this machine reports how much life the installed list has left. dev01 has
# /healthz because APP-01 is an HTTP service that was already being polled.
# PostgreSQL is not, so the same question has to be asked from outside, by
# hand, with crl-status. OT-040 on a second machine.
SHELL=/bin/sh
PATH=/usr/local/bin:/usr/bin:/bin
*/30 * * * * fetch-crl --url http://pub01.lab.simurgh.example/crl.pem --anchors /etc/postgresql/15/main/ca-bundle.pem --install /var/lib/postgresql/crl/crl.pem --state /var/lib/postgresql/crl/state.json >>/var/lib/postgresql/crl/fetch.out 2>&1
The database now has a dependency it did not have this morning. If that list goes stale,
PostgreSQL refuses every client, not merely revoked ones, which is Chapter 09's measurement
arriving on a machine with a much larger blast radius than dev01.
sudo docker exec -i db01 sh <<'OUTER'
cat >> /etc/postgresql/15/main/postgresql.conf <<'CONF'
# Chapter 12: db01 verifies its clients.
ssl_ca_file = '/etc/postgresql/15/main/ca-bundle.pem'
ssl_crl_file = '/var/lib/postgresql/crl/crl.pem'
CONF
tail -4 /etc/postgresql/15/main/postgresql.conf
OUTER
sudo docker exec db01 pg_ctlcluster 15 main restart
sudo docker exec db01 su postgres -c "psql -tAc \"SHOW ssl_ca_file\""
sudo docker exec db01 su postgres -c "psql -tAc \"SHOW ssl_crl_file\""Expected: the two lines as written, then the two paths.
Two quoted heredocs and docker exec -i, and each part is load-bearing. PostgreSQL wants
single quotes around a path, which is awkward inside a sh -c '...' and invites repairing the
quoting afterwards with a global sed. A global substitution on a configuration file is a
disproportionate tool: it edits lines Chapters 04 and 05 put there and any comment that happens
to contain the character. Quoting the heredoc delimiter stops the shell interpolating anything,
so the lines land exactly as written and nothing else in the file is touched.
-i is there because a heredoc is stdin, and Chapter 02 §4 lost time to a docker exec without
it: the command runs, reads nothing, and reports success.
Nothing has changed for the application yet. pg_hba still says scram-sha-256, so the
password path still works and the certificate is not being asked for. That is deliberate: the
verifier is switched on before anything depends on it, so a mistake here is discovered while the
old path still functions.
curl -s http://127.0.0.1:8080/payments/1001/statusExpected: the payment record, still authenticated by password.
Before switching pg_hba, understand exactly what PostgreSQL will check, because two of the
three checks are not where you would look for them.
Ask ca01 for a certificate naming the database, which POL-02 has permitted since Chapter
07, and try to log in as paymentsvc with it:
sudo docker exec -u paymentsvc dev01 sh -c '
openssl ecparam -name prime256v1 -genkey -noout -out /tmp/wrong.key
chmod 0400 /tmp/wrong.key
openssl req -new -key /tmp/wrong.key -out /tmp/wrong.csr \
-subj "/CN=db01.lab.simurgh.example"'
sudo docker cp dev01:/tmp/wrong.csr /tmp/wrong.csr
sudo docker cp /tmp/wrong.csr ca01:/opt/ca-client/requests/wrong.csr
sudo docker exec ca01 chown ca:ca /opt/ca-client/requests/wrong.csr
sudo docker exec -u ca ca01 request-cert /opt/ca-client/requests/wrong.csr \
db01.lab.simurgh.example | head -2
sudo docker cp ca01:/opt/ca-client/issued/db01.lab.simurgh.example.chain.crt /tmp/wrong.crt
sudo docker cp /tmp/wrong.crt dev01:/tmp/wrong.crtExpected: an issued certificate. POL-02 allowed this, because ca01 is permitted to
request the database's name and always has been.
Switch pg_hba to certificate authentication and try it:
sudo docker exec db01 sh -c '
cp /etc/postgresql/15/main/pg_hba.conf /etc/postgresql/15/main/pg_hba.conf.scram
sed -i "s|^hostssl.*scram-sha-256|hostssl paymentsdb paymentsvc all cert clientcert=verify-full|" \
/etc/postgresql/15/main/pg_hba.conf
grep -v "^#" /etc/postgresql/15/main/pg_hba.conf | grep -v "^$"'
sudo docker exec db01 pg_ctlcluster 15 main reload
sudo docker exec db01 su postgres -c "psql -tAc \"ALTER ROLE paymentsvc LOGIN\""
sudo docker exec -u paymentsvc dev01 sh -c '
PGSSLCERT=/tmp/wrong.crt PGSSLKEY=/tmp/wrong.key \
PGSSLROOTCERT=/opt/paymentsvc/ca.crt \
psql "host=db01.lab.simurgh.example dbname=paymentsdb user=paymentsvc sslmode=verify-full" \
-tAc "select 1"' 2>&1 | tail -2Expected:
psql: error: connection to server at "db01.lab.simurgh.example" (172.x.x.x), port 5432 failed:
FATAL: certificate authentication failed for user "paymentsvc"
Two gates said yes and one said no, and they were answering different questions. POL-02
decided whether ca01 may request that name. db01 decided whether this certificate may
be paymentsvc. An estate that only had the first would let any certificate its own
authority ever issued log in as anything.
That is Chapter 09 §1's lesson pointed at the database, and this time the answer is a refusal rather than an incident.
Figure 12.2 — three checks, and only one of them is authentication
flowchart TB
A["client connects, presents CERT-11 and its chain"]:::app
B{"1. TLS: does the chain<br/>reach ssl_ca_file?"}:::control
C{"2. TLS: is any certificate<br/>in it on ssl_crl_file?"}:::control
D{"3. pg_hba: is there a rule<br/>for this host, user, database?"}:::control
E{"4. cert auth: does the CN<br/>equal the requested role?"}:::control
F["connected as paymentsvc"]:::best
A --> B
B -->|no| X1["handshake fails<br/>certificate verify failed"]:::retired
B -->|yes| C
C -->|revoked| X2["★ handshake fails<br/>sslv3 alert certificate revoked"]:::retired
C -->|clean| D
D -->|no rule| X3["FATAL: no pg_hba.conf entry"]:::retired
D -->|cert| E
E -->|no| X4["★ FATAL: certificate<br/>authentication failed for user"]:::retired
E -->|yes| F
classDef app fill:#e5e7eb,stroke:#6b7280,stroke-width:1px,color:#111827
classDef control fill:#dbeafe,stroke:#1d4ed8,stroke-width:2px,color:#0b1f4b
classDef best fill:#dcfce7,stroke:#15803d,stroke-width:3px,color:#052e16
classDef retired fill:#f8fafc,stroke:#cbd5e1,stroke-width:1px,stroke-dasharray:2 4,color:#94a3b8
Steps 1 and 2 happen in the TLS handshake, before pg_hba is consulted at all. That was
measured, and it was measured by accident: a spike with a broken pg_hba still returned sslv3 alert certificate revoked, which places the revocation check unambiguously.
The consequence is worth stating plainly. Revocation on this server cannot be weakened by
editing an authentication rule. Somebody who changes cert back to scram-sha-256 has not
disabled revocation checking; they have only changed what happens after it passes. That is an
unusually good property and it is the opposite of where most people would look for it.
Deploy the application that has no password:
# /opt/paymentsvc/config.yaml
database:
host: db01.lab.simurgh.example
port: 5432
name: paymentsdb
sslmode: verify-full
# Chapter 12. The role this process logs in as, and there is no password
# anywhere in this file or in the secret store to go with it.
#
# This name has to equal the Common Name of the certificate below.
# PostgreSQL's `cert` authentication takes the CN and requires it to match
# the role being requested; measured, a valid certificate naming anything
# else is refused with `certificate authentication failed for user`.
#
# Note what the name is NOT. Every other certificate in this estate is named
# for a machine. This one is named for a workload, because the thing being
# authenticated is the application and not the host it happens to run on.
user: paymentsvc
# CERT-11 and its key. The certificate is the chain, leaf followed by
# CERT-09, for the reason every other holder here presents a chain: the
# server can build the path itself today, and that depends on somebody
# else's configuration staying right.
#
# NOT in /opt/paymentsvc, and Chapter 10 learned why the hard way. That
# directory is root-owned so that APP-01 cannot rewrite its own
# configuration, which has been true since Chapter 01. Creating a key
# there, and replacing it during an incident, both need write permission
# on the DIRECTORY and not merely on the file. So the identity lives
# where ACC-03 owns the directory, exactly as the CRL does.
sslcert: /var/lib/paymentsvc/client.crt
sslkey: /var/lib/paymentsvc/client.key
# Chapter 05: the anchor is the authority, not the server.
#
# This was /opt/paymentsvc/db01.crt, a copy of the certificate db01
# presents. Pinning that meant re-issuing db01's certificate broke this
# client, which is OT-017. Now it is CERT-02, the root, and db01 can be
# re-issued as often as it likes without this line or this file changing.
#
# The path is the only thing that had to change on the client. That is
# the whole payoff, and it is a one-time cost.
sslrootcert: /opt/paymentsvc/ca.crt
# Chapter 09: check whether the certificate has been taken back.
#
# The anchor above answers "did our authority sign this". It cannot answer
# "does our authority still stand behind it", and those became different
# questions the moment a credential was stolen. This line is the second
# question.
#
# It is not a free improvement. With this set, libpq refuses any
# certificate whose revocation status it cannot establish: no file, an
# unreadable file, or a list past its nextUpdate all stop this application
# from starting, healthy certificates included. Deleting the line turns
# revocation checking off and everything works again, which is exactly why
# it is worth knowing that the line is load-bearing.
#
# Chapter 10: moved out of this directory, and the reason is least privilege
# rather than tidiness. The list is now maintained by fetch-crl, running as
# `paymentsvc`, and an atomic replace needs a temporary file in the same
# directory as the target. Granting that here would make /opt/paymentsvc
# writable by the application, which would let APP-01 rewrite its own
# configuration, and that has been forbidden since Chapter 01.
#
# A revocation list is not configuration. It is state an agent maintains, so
# it lives where the agent can own it.
sslcrl: /var/lib/fetch-crl/crl.pem
# Chapter 12: still configured, and no longer used for the database.
#
# SVC-02 has no consumer as of this chapter. That is not a reason to delete
# it, and the chapter says why: a store with nothing in it is the correct
# outcome of removing the thing it was holding, and the next secret this
# estate acquires will want somewhere to live. What it does mean is that
# OT-011, a single point of total compromise holding everything in
# plaintext, currently holds nothing worth compromising.
secret_store:
socket: /run/secretstore/sock
secret_name: paymentsvc-db
server:
listen: 0.0.0.0:8080
# Chapter 11. How close to the deadline counts as a problem.
#
# Two days against the intermediate's seven means two consecutive missed
# refreshes are needed before /healthz goes amber, so a single missed run is
# not an incident and a broken pipeline is. Set it too low and the alarm
# arrives at the same time as the outage, which is the same as no alarm.
crl:
warn_days: 2#!/usr/bin/env python3
"""APP-01 paymentsvc, answers 'what is the status of payment X?'
Chapter 04 change: SVC-01 now lives on its own machine, so the database
connection crosses a network. It is made with TLS and the server
certificate is verified against a pinned copy, which is what stops anything
that answers on port 5432 from being handed our queries.
The credential still comes from SVC-02 over a Unix socket on this host, and
this process still stores no credential of its own.
"""
import http.client
import json
import logging
import os
import pwd
import socket
import subprocess
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import psycopg2
import yaml
from psycopg2.extras import RealDictCursor
CONFIG_PATH = os.environ.get("PAYMENTSVC_CONFIG", "/opt/paymentsvc/config.yaml")
logging.basicConfig(
level=os.environ.get("LOG_LEVEL", "INFO").upper(),
format="%(asctime)s %(levelname)s %(message)s",
handlers=[
logging.FileHandler("/var/log/paymentsvc.log"),
logging.StreamHandler(sys.stdout),
],
)
log = logging.getLogger("paymentsvc")
def load_config(path):
log.info("loading configuration from %s", path)
with open(path) as fh:
cfg = yaml.safe_load(fh)
log.debug("effective configuration: %s", cfg)
return cfg
class UnixHTTPConnection(http.client.HTTPConnection):
"""An HTTP client that speaks over a Unix domain socket.
The only difference from an ordinary HTTPConnection is where connect()
points. Everything above it, requests, status codes, headers, is
unchanged, which is why moving SVC-02 off TCP cost so little here.
"""
def __init__(self, socket_path, timeout=5):
super().__init__("localhost", timeout=timeout)
self.socket_path = socket_path
def connect(self):
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.settimeout(self.timeout)
sock.connect(self.socket_path)
self.sock = sock
def fetch_credential(socket_path, secret_name):
"""Ask SVC-02 for the current database credential.
We present no token and assert no identity. The kernel tells the store
our uid when we connect, and the store applies POL-01 to it. If it says
no, we get a 403 and fail loudly, being refused is not something to
paper over.
"""
conn = UnixHTTPConnection(socket_path)
try:
conn.request("GET", f"/v1/secrets/{secret_name}")
resp = conn.getresponse()
body = resp.read().decode()
if resp.status == 403:
raise PermissionError(
f"secretstore refused this process: {json.loads(body).get('detail')}"
)
if resp.status != 200:
raise RuntimeError(f"secretstore returned {resp.status}: {body}")
payload = json.loads(body)
finally:
conn.close()
cred = json.loads(payload["value"])
log.info("fetched %s version %s as user %s",
secret_name, payload["version"], cred["user"])
return cred["user"], cred["password"], payload["version"]
def check_crl_usable(path):
"""Refuse to start rather than run with revocation checking silently off.
THIS FUNCTION EXISTS BECAUSE libpq FAILS OPEN, AND IT WAS MEASURED.
Naming a CRL is supposed to turn revocation checking on. libpq only turns
it on if the file loads; if it cannot, the flags are never set, the
connection succeeds, and a revoked certificate is accepted exactly as
before. Four ways of being unusable were tested against PostgreSQL 15 and
all four connected: the file missing, the file unreadable, the file
containing something that is not a CRL, and the file empty.
None of them warned. The last is the one that happens in real life: a
fetch that failed and left a zero-byte file behind, after which the
estate stops checking revocation and nothing anywhere says so.
So the application checks on the platform's behalf, and refuses to start
when the answer is no. That is D-011: a service configured to require a
protection should fail loudly rather than run without it. A crash at
startup is a page; a silently disabled security control is a year of
believing something that is not true.
Note what is NOT checked here: whether the file carries a list from every
authority in the chain. libpq fails CLOSED on that one, refusing healthy
certificates, so it is already loud and needs no help.
"""
if not os.path.exists(path):
raise SystemExit(f"sslcrl is set to {path}, which does not exist. "
"Refusing to start: revocation checking would be off.")
if not os.access(path, os.R_OK):
raise SystemExit(f"sslcrl is set to {path}, which is not readable by "
f"{pwd.getpwuid(os.getuid()).pw_name}. Refusing to start.")
if os.path.getsize(path) == 0:
raise SystemExit(f"sslcrl is set to {path}, which is empty. "
"Refusing to start: a failed download looks exactly like this.")
# Parse every list in the file and report the issuers, so the log says
# what is actually being enforced rather than that a setting is spelled.
proc = subprocess.run(["openssl", "crl", "-in", path, "-noout", "-issuer"],
capture_output=True, text=True)
if proc.returncode != 0:
raise SystemExit(f"sslcrl is set to {path}, which openssl cannot parse. "
"Refusing to start.")
log.info("CRL file %s parses, first issuer %s", path, proc.stdout.strip())
class Database:
"""Owns the connection and the credential it was made with."""
def __init__(self, cfg):
self.cfg = cfg
self.conn = None
self.user = None
self.version = None
self.connect()
def connect(self):
"""Open the connection. CHAPTER 12: WITH NO PASSWORD AT ALL.
Everything above this line about fetching a credential is still here,
still correct, and no longer used for the database. SEC-02 and SEC-03
are retired: what proves who this process is now is a private key it
holds and a certificate our own authority signed, and the database
checks both at the TLS layer before it looks at any authentication
rule.
Read what has NOT improved, because it is the honest half. There is
still exactly one durable secret on this host, it still lives for the
life of the process, and it is still readable by root. It changed
shape from a password fetched at startup into a private key read at
startup, which is a different object with the same lifecycle. OT-006
is not closed by this chapter, it is renamed.
What DID improve is that the secret is no longer shared. A password
is a thing two parties know; a private key is a thing one party has.
The database has never seen this key and cannot leak it, which is the
property no amount of rotation could buy.
"""
db = self.cfg["database"]
user = db["user"]
conn_args = dict(
host=db["host"], port=db["port"], dbname=db["name"],
user=user,
# No password= at all. Not an empty one: absent. If pg_hba on the
# far side ever falls back to a password method, this connection
# fails rather than quietly negotiating something weaker, which
# is the same argument as sslmode=verify-full over require.
sslmode=db["sslmode"],
# The identity. sslcert is presented during the handshake and
# PostgreSQL matches its Common Name against the role named
# above; measured, they must agree or it refuses with
# `certificate authentication failed for user "..."`.
sslcert=db["sslcert"], sslkey=db["sslkey"],
)
# The anchor is optional, and leaving it out is not neutral. libpq
# verifies the server certificate whenever a root CA file is
# present, even under sslmode=require, so naming one here makes a
# weak-looking config stronger than it reads. Section 7 is what
# happens when the line is missing and nobody noticed it mattered.
if db.get("sslrootcert"):
conn_args["sslrootcert"] = db["sslrootcert"]
# Chapter 09. Naming a CRL turns revocation checking ON, and that is
# not a small switch: libpq then refuses any certificate it cannot
# check, not merely the ones that were revoked. A missing file, an
# unreadable one, or a list whose nextUpdate has passed all stop this
# application from starting. That is correct and it is expensive, and
# section 7 is what it looks like when nobody refreshed the list.
if db.get("sslcrl"):
check_crl_usable(db["sslcrl"])
conn_args["sslcrl"] = db["sslcrl"]
# The key file's mode is checked by libpq, not by us, and the error
# is one of the clearest in this build: `private key file ... has
# group or world access; file must have permissions u=rw (0600) or
# less`. Nothing here needs to duplicate it.
self.conn = psycopg2.connect(**conn_args)
self.conn.autocommit = True
self.user, self.version = user, "certificate"
log.info("connected to %s@%s:%s/%s (auth %s, sslmode %s, crl %s)",
user, db["host"], db["port"], db["name"], "certificate",
db["sslmode"], "on" if db.get("sslcrl") else "off")
def query(self, sql, args):
"""Run a query; on a connection-level failure, re-fetch and retry once."""
try:
return self._query(sql, args)
except psycopg2.OperationalError as exc:
log.warning("connection failed (%s), re-fetching credential", exc)
self.connect()
return self._query(sql, args)
def _query(self, sql, args):
with self.conn.cursor(cursor_factory=RealDictCursor) as cur:
cur.execute(sql, args)
return cur.fetchone()
cfg = load_config(CONFIG_PATH)
database = Database(cfg)
class Handler(BaseHTTPRequestHandler):
server_version = "paymentsvc/0.4"
def _json(self, code, payload):
body = json.dumps(payload).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_GET(self):
if self.path == "/healthz":
# CHAPTER 11: THIS ENDPOINT FINALLY ANSWERS A QUESTION.
#
# Until now it returned a hardcoded ok, and the book complained
# about it twice: Chapter 08 section 10 found it reporting healthy
# while the application could not reach its database at all, and
# Chapter 10 section 0 warned against using it as a state check.
#
# It is not being given a database probe. That was the right call
# and stays: a health endpoint that opens a connection turns every
# poll into load, and Chapter 08's complaint was that it claimed
# more than it knew, not that it knew too little.
#
# What it reports now is the one thing about this process that
# degrades silently and takes the estate down with it: how much
# life is left in the revocation list. Measured in Chapter 11, a
# cron job that stops working leaves no trace anywhere, so the job
# cannot be the thing that is watched. The artefact can.
#
# It shells out to crl-status rather than reimplementing it, for
# the reason SVC-03 shells out to sign-leaf: two copies of a rule
# about expiry is one copy too many, and the one that drifts will
# be the one nobody runs by hand.
crl = cfg["database"].get("sslcrl")
if not crl:
return self._json(200, {"status": "ok", "crl_checking": False})
proc = subprocess.run(
["/usr/local/bin/crl-status", "--crl", crl,
"--warn-days", str(cfg.get("crl", {}).get("warn_days", 2))],
capture_output=True, text=True)
detail = [ln for ln in proc.stdout.splitlines() if ln.strip()]
if proc.returncode == 0:
return self._json(200, {"status": "ok", "crl": detail})
# 503, not 200 with a flag. A monitoring system reads the status
# line; a field inside a 200 is a thing somebody has to remember
# to look at, which is the failure this whole chapter is about.
return self._json(503, {"status": "degraded", "crl": detail})
if self.path == "/credinfo":
return self._json(200, {
"db_user": database.user,
"secret_name": cfg["secret_store"]["secret_name"],
# Chapter 12: there is no credential version any more,
# because there is no credential. What identifies this
# process is a certificate, so report that instead.
"auth_method": database.version,
"client_cert": cfg["database"].get("sslcert"),
"running_as": pwd.getpwuid(os.getuid()).pw_name,
"uid": os.getuid(),
"db_host": cfg["database"]["host"],
"sslmode": cfg["database"]["sslmode"],
# Chapter 09: whether this client is checking revocation at
# all. Worth exposing, because the difference between
# checking and not checking is invisible from the outside
# and is the difference between refusing a revoked
# certificate and accepting one.
# Effect, not intent. Reporting that a setting is spelled
# would be the Chapter 08 /healthz defect again: with an
# unusable file libpq checks nothing while the config still
# says sslcrl. The process refuses to start in that case, so
# if this says true it is true.
"crl_checking": bool(cfg["database"].get("sslcrl")),
})
parts = self.path.strip("/").split("/")
if len(parts) == 3 and parts[0] == "payments" and parts[2] == "status":
try:
payment_id = int(parts[1])
except ValueError:
return self._json(400, {"error": "payment id must be an integer"})
row = database.query(
"SELECT id, reference, amount_cents, currency, status "
"FROM payments WHERE id = %s",
(payment_id,),
)
if row is None:
return self._json(404, {"error": "no such payment"})
return self._json(200, dict(row))
return self._json(404, {"error": "not found"})
def log_message(self, fmt, *args):
log.info("%s %s", self.address_string(), fmt % args)
if __name__ == "__main__":
host, _, port = cfg["server"]["listen"].rpartition(":")
srv = ThreadingHTTPServer((host, int(port)), Handler)
log.info("listening on %s", cfg["server"]["listen"])
srv.serve_forever()sudo docker cp dev01/app/config.yaml dev01:/opt/paymentsvc/config.yaml
sudo docker cp dev01/app/paymentsvc.py dev01:/opt/paymentsvc/paymentsvc.py
sudo docker exec dev01 sh -c '
chown paymentsvc:paymentsvc /opt/paymentsvc/config.yaml /opt/paymentsvc/paymentsvc.py
chmod 0400 /opt/paymentsvc/config.yaml
chmod 0444 /opt/paymentsvc/paymentsvc.py'
sudo docker exec dev01 pkill -f 'python3 /opt/paymentsvc/paymentsvc.py' || true
sudo docker exec -d -u paymentsvc dev01 \
sh -c 'python3 /opt/paymentsvc/paymentsvc.py >>/var/log/paymentsvc.out 2>&1'
sleep 2
curl -s http://127.0.0.1:8080/payments/1001/status
curl -s http://127.0.0.1:8080/credinfo; echoExpected: the payment record, and "db_user": "paymentsvc", "auth_method": "certificate",
"client_cert": "/var/lib/paymentsvc/client.crt".
Confirm from the database's side, which is the view that cannot be faked by the client:
sudo docker exec db01 su postgres -c \
"psql -tAc \"SELECT usename, ssl, client_dn, issuer_dn FROM pg_stat_ssl \
JOIN pg_stat_activity USING (pid) WHERE usename = 'paymentsvc'\""Expected: paymentsvc|t|CN=paymentsvc|CN=Simurgh Lab Issuing CA 1.
The database is recording who connected and which authority vouched for them, which is
something a password connection could never tell it. SVC-02 has logged consumer identity since
Chapter 03; this is the first time SVC-01 can.
sudo docker exec dev01 sh -c 'chmod 0644 /var/lib/paymentsvc/client.key'
sudo docker exec dev01 pkill -f 'python3 /opt/paymentsvc/paymentsvc.py' || true
sudo docker exec -u paymentsvc dev01 python3 /opt/paymentsvc/paymentsvc.py 2>&1 | tail -3
sudo docker exec dev01 sh -c 'chmod 0400 /var/lib/paymentsvc/client.key'Expected, ending in a traceback whose last line is:
psycopg2.OperationalError: connection to server at "db01.lab.simurgh.example" (172.x.x.x),
port 5432 failed: private key file "/var/lib/paymentsvc/client.key" has group or world access;
file must have permissions u=rw (0600) or less if owned by the current user, or permissions
u=rw,g=r (0640) or less if owned by root
Read where that sentence changes speaker. The first half is psycopg2 reporting that a
connection failed. The second half is libpq, which never asked the server anything: it looked at
the key file, decided it was unfit to use, and reported the refusal in the shape of a connection
error. db01 was not consulted and its log has nothing to say about this.
That is one of the clearest errors in this build, and it is worth noticing what it does not do: it does not warn and continue. A private key that others can read is treated as a configuration error rather than a risk to be accepted.
Which makes libpq's inconsistency worth naming. It fails closed and loudly on a key file with
the wrong mode, and Chapter 09 measured it failing open and silently on a CRL file it cannot
load. Same library, same category of input, opposite behaviour, and the reason check_crl_usable
exists is that the second case had to be written by hand. Nothing in the documentation of either
prepares you for the difference.
Restart the application:
sudo docker exec -d -u paymentsvc dev01 \
sh -c 'python3 /opt/paymentsvc/paymentsvc.py >>/var/log/paymentsvc.out 2>&1'
sleep 2
curl -s http://127.0.0.1:8080/payments/1001/statusExpected: the payment record.
The estate has been able to revoke certificates since Chapter 09 and it has never protected anything the business depends on. It does now.
sudo docker cp dev01:/var/lib/paymentsvc/client.crt /tmp/appcert.crt
sudo docker cp /tmp/appcert.crt hsm01:/var/lib/ca/requests/appcert.crt
sudo docker exec hsm01 chown signd:signd /var/lib/ca/requests/appcert.crt
sudo docker exec -u signd hsm01 sh -c '
awk "/BEGIN/{n++} n==1" /var/lib/ca/requests/appcert.crt > /var/lib/ca/requests/appleaf.crt'
sudo docker exec -u signd hsm01 \
revoke-cert /var/lib/ca/requests/appleaf.crt keyCompromise | tail -4Expected: the republished list, and one more revoked entry.
Push it through the pipeline to the machine that will act on it:
sudo docker exec -u pub pub01 pull-artifacts --from http://hsm01.lab.simurgh.example:8080 --once
sudo docker exec -u postgres db01 fetch-crl \
--url http://pub01.lab.simurgh.example/crl.pem \
--anchors /etc/postgresql/15/main/ca-bundle.pem \
--install /var/lib/postgresql/crl/crl.pem \
--state /var/lib/postgresql/crl/state.json | head -1
sudo docker exec db01 pg_ctlcluster 15 main reload
sleep 1
sudo docker exec dev01 pkill -f 'python3 /opt/paymentsvc/paymentsvc.py' || true
sudo docker exec -u paymentsvc dev01 python3 /opt/paymentsvc/paymentsvc.py 2>&1 | tail -3Expected, ending in:
psycopg2.OperationalError: connection to server at "db01.lab.simurgh.example" (172.x.x.x),
port 5432 failed: SSL error: sslv3 alert certificate revoked
sslv3 alert certificate revoked, and read where it came from. That is the server
speaking during the handshake. The client was not consulted, pg_hba was not reached, and no
authentication rule was involved. db01 was told this certificate had been withdrawn and refused
to continue.
This is the first time in twelve chapters that revoking something has protected the application path. Chapters 09, 10 and 11 built the register, the publication point, the agent and the clock, and every one of them protected the authority's own connections. The work only becomes valuable at the moment a second verifier uses it.
OT-037 is half closed: two of three verifiers now check. SVC-03 still does not.
Issue a replacement, which is what an incident actually looks like:
sudo docker exec -u paymentsvc dev01 sh -c '
openssl ecparam -name prime256v1 -genkey -noout -out /var/lib/paymentsvc/client.key.new
chmod 0400 /var/lib/paymentsvc/client.key.new
openssl req -new -key /var/lib/paymentsvc/client.key.new \
-out /tmp/paymentsvc2.csr -subj "/CN=paymentsvc"'
sudo docker cp dev01:/tmp/paymentsvc2.csr /tmp/paymentsvc2.csr
sudo docker cp /tmp/paymentsvc2.csr ca01:/opt/ca-client/requests/paymentsvc.csr
sudo docker exec ca01 chown ca:ca /opt/ca-client/requests/paymentsvc.csr
sudo docker exec -u ca ca01 request-cert --client /opt/ca-client/requests/paymentsvc.csr paymentsvc | head -1
sudo docker cp ca01:/opt/ca-client/issued/paymentsvc.chain.crt /tmp/client2.crt
sudo docker cp /tmp/client2.crt dev01:/var/lib/paymentsvc/client.crt
sudo docker exec dev01 sh -c '
mv /var/lib/paymentsvc/client.key.new /var/lib/paymentsvc/client.key
chown paymentsvc:paymentsvc /var/lib/paymentsvc/client.crt /var/lib/paymentsvc/client.key
chmod 0444 /var/lib/paymentsvc/client.crt
chmod 0400 /var/lib/paymentsvc/client.key'
sudo docker exec -d -u paymentsvc dev01 \
sh -c 'python3 /opt/paymentsvc/paymentsvc.py >>/var/log/paymentsvc.out 2>&1'
sleep 2
curl -s http://127.0.0.1:8080/payments/1001/statusExpected: the payment record.
A new key, a new certificate, the same name, and nothing on db01 changed. No role was
altered, no password was set, no rotation procedure ran and no second identity was needed. The
database's configuration is a statement about who may connect, and it did not have to know that
the answer had been re-issued.
Four objects and one procedure stop being needed. Retire them properly rather than leaving them lying around:
sudo docker exec db01 su postgres -c "psql -q -c \"DROP ROLE IF EXISTS paymentsvc_a\""
sudo docker exec db01 su postgres -c "psql -q -c \"DROP ROLE IF EXISTS paymentsvc_b\""
sudo docker exec db01 su postgres -c \
"psql -tAc \"SELECT rolname, rolcanlogin FROM pg_authid WHERE rolname LIKE 'paymentsvc%'\""Expected: one row, paymentsvc|t.
| Retired | Was | Why it is gone |
|---|---|---|
SEC-02, SEC-03 |
the two rotating database passwords | there is no shared secret to rotate |
ACC-05, ACC-06 |
paymentsvc_a and paymentsvc_b |
they existed only so one could be changed while the other served |
PROC-01 |
the six-step zero-downtime rotation | replaced by issuing a certificate, which needs no coordination at all |
Chapter 02 was not wrong. Given a shared password, every step of PROC-01 was necessary, and
the two-role dance is the correct answer to "how do I change a value two parties must agree on,
without a window where neither works". The complexity was real and it was in service of a
constraint.
Remove the constraint and the complexity evaporates. That is the more valuable lesson, and it
is one this build could only teach by living with the password for eleven chapters first. A
reader told in Chapter 02 that certificates would be simpler would have learned a slogan. A
reader who built PROC-01, ran it, verified convergence, and then watched all of it become
unnecessary has learned something about where complexity comes from.
SVC-02 now has no consumers. It is still running, still holds POL-01, and holds nothing
the estate needs. That is not a reason to delete it, and it does mean OT-011, a single point of
total compromise holding everything in plaintext, currently holds nothing worth compromising. The
next secret this estate acquires will want somewhere to live.
Bought. The database and the application no longer share a value. SVC-01 records who
connected and which authority vouched for them. A compromised credential can be withdrawn and the
withdrawal is enforced by the server during the handshake, where no authentication rule can
weaken it. Replacing the credential is one command and touches nothing on the database.
Not bought, and the first of these is the important one.
OT-006 is renamed, not closed. There is still exactly one durable secret on HOST-01, it
is still read at startup and held for the life of the process, and root still reads it. A
password became a private key: a different object with the same lifecycle. Nothing here fetches a
credential at run time, uses it, and discards it. OT-006 stays open and is now the clearest gap
in the build.
The separation between workloads is a file mode. 0400 owned by ACC-03 is what stops
SVC-02 becoming paymentsvc. That is OT-004, untouched since Chapter 01.
db01 acquired a dependency it cannot report on. If its CRL goes stale, PostgreSQL refuses
every client at once. dev01 has /healthz because APP-01 was already being polled;
PostgreSQL is not, so the same question has to be asked by hand. OT-041.
POL-02 now mixes hostnames and workload names with nothing distinguishing them, and one
rule grants both. OT-016 and OT-023, on a policy that has just become more powerful.
And SVC-03 still checks no revocation. The authority is the last verifier that does not.
OT-037, half closed.
Figure 12.3 — after Chapter 12
flowchart LR
subgraph OFF["HOST-05 rootca: no network, Exited"]
ROOT[["KEY-05, CERT-08"]]:::crypto
end
subgraph LAB["NET-01 lab, one flat network: OT-024"]
direction TB
subgraph H4["HOST-04 hsm01"]
SIGND("SVC-03 signd<br/>POL-02 now names a workload"):::control
ICA[["KEY-06, CERT-09"]]:::crypto
end
subgraph H6["HOST-06 pub01: holds nothing"]
PUBD("SVC-04 pubd"):::control
end
subgraph H1["HOST-01 dev01"]
APP["APP-01 paymentsvc<br/>★ no password anywhere"]:::app
CK{{"★ CERT-11 + key, 0400<br/>the only durable secret left"}}:::secret
STORE("SVC-02 secretstore<br/>★ no consumers"):::retired
end
subgraph H2["HOST-02 db01"]
DB[("SVC-01 paymentsdb")]:::store
VER("★ verifies clients<br/>ssl_ca_file + ssl_crl_file"):::control
DCRL{{"★ its own CRL, on a clock"}}:::secret
end
subgraph H3["HOST-03 ca01"]
OPS(["ACC-01 you"]):::human
end
end
ROOT ==>|"signs once"| ICA
ICA ==>|"signs CERT-11"| CK
OPS -->|"request-cert paymentsvc"| SIGND
PUBD -.->|"CRL-01"| DCRL
PUBD -.->|"CRL-01"| APP
CK ==>|"★ mTLS: the app proves who it is"| VER
VER --> DB
DCRL -.->|"★ revoked clients refused<br/>during the handshake"| VER
style OFF fill:#ecfdf5,stroke:#15803d,stroke-width:3px,stroke-dasharray:8 4
style LAB fill:#f8fafc,stroke:#475569,stroke-width:1px,stroke-dasharray:6 3
style H1 fill:#ffffff,stroke:#475569,stroke-width:1px
style H2 fill:#ffffff,stroke:#475569,stroke-width:1px
style H3 fill:#ffffff,stroke:#475569,stroke-width:1px
style H4 fill:#fff7ed,stroke:#b45309,stroke-width:2px,stroke-dasharray:6 3
style H6 fill:#f1f5f9,stroke:#94a3b8,stroke-width:1px
classDef human fill:#ffffff,stroke:#4b5563,stroke-width:2px,stroke-dasharray:4 3,color:#111827
classDef app fill:#e5e7eb,stroke:#6b7280,stroke-width:1px,color:#111827
classDef control fill:#dbeafe,stroke:#1d4ed8,stroke-width:2px,color:#0b1f4b
classDef crypto fill:#fee2e2,stroke:#b91c1c,stroke-width:4px,color:#450a0a
classDef store fill:#e2e8f0,stroke:#475569,stroke-width:1px,color:#0f172a
classDef secret fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#451a03
classDef retired fill:#f8fafc,stroke:#cbd5e1,stroke-width:1px,stroke-dasharray:2 4,color:#94a3b8
SVC-02 is drawn dotted for the first time. It has not been removed and has nothing to do,
which is what happens to a component when the problem it existed for is solved somewhere else.
HOST-02 gained a blue box. The database decides something now: whether the certificate in
front of it is one this estate still stands behind. It is the second verifier in the estate and
the first one whose refusals protect the business.
The thick edge from the application to the database is the one that mattered from Chapter 01 and it finally carries an identity rather than a shared value.
Six machines. The application authenticates to its database with a certificate naming the
workload rather than the host, issued by an intermediate whose root is switched off, checked by a
database that verifies the chain and the revocation list during the handshake. No password exists
anywhere in the path. One durable secret remains on HOST-01, held for the life of the process,
and nothing fetches, uses and discards anything.
| ID | Decision |
|---|---|
D-084 |
The certificate names the workload, not the host |
D-085 |
db01 verifies clients, which makes it the estate's second verifier |
D-086 |
One login role, not two |
D-087 |
SVC-02 stays, with nothing in it |
D-084, and why it matters more than it looks. A host certificate is shared by every process
on that host, limited only by file permissions, which is exactly the argument Chapter 03 made
about SEC-01. Naming the workload makes the certificate mean what it says. The cost is that the
separation is still a file mode, and the benefit is that this is the same shift Kubernetes
forces, met now with a mechanism we understand.
D-085, and the property that came free. Measured: ssl_crl_file is enforced during the TLS
handshake, before pg_hba is consulted. So revocation on this server cannot be weakened by
editing an authentication rule, which is the opposite of where anybody would look for that
guarantee. The cost is a new hard dependency: a stale CRL on db01 refuses every client at once.
D-086, why PROC-01 could be retired rather than adapted. Two login roles existed so that
one could be changed while the other served, which is the only way to rotate a value two parties
must agree on. Certificates for the same name can overlap in validity without the database being
told anything, so the second identity has nothing left to do. Chapter 02 was not wrong; its
constraint was removed.
OT-006, renamed and now the clearest gap in the build. A password held for the life of the
process became a private key held for the life of the process. Nothing here fetches a short-lived
credential at run time, uses it and discards it, which is the property that would make this
estate's story about secrets complete, and the one that has to exist before Kubernetes can be
shown to lack it.
OT-041 — db01 cannot report on the dependency it just acquired. Its CRL must stay fresh
or every client is refused, and there is no endpoint to ask. crl-status is installed there and
nothing runs it for anybody's benefit. OT-040 on a second machine, with a larger blast radius.
OT-037, half closed. APP-01 and SVC-01 check revocation. SVC-03, the authority
itself, does not, so the stolen client credential from Chapter 09 §1 still opens a connection to
the thing that issues certificates.
OT-016 and OT-023 — POL-02 now grants a workload name and a hostname from one rule,
with nothing in the file marking which is which, no reason recorded and no expiry. The policy
became more powerful and no more reviewable.
OT-004, untouched and now load-bearing. The separation between APP-01 and every other
process on HOST-01 is 0400 on one file. Root reads it.
OT-020, and the pass that follows this chapter. Three chapters have installed software into
running containers. The Dockerfiles describe a system that has not existed since Chapter 09.
- Looked at what the password actually cost: not weakness, but the fact that two parties must
agree on a value, and every part of
PROC-01existed to make that survivable. - Established the difference between a certificate naming a host and one naming a workload, and that the second is what Kubernetes will require.
- Asked the authority for a workload name and were refused by
POL-02, correctly, for the first time. - Made
db01a verifier: anchors, a revocation list, an agent and a clock, at the cost of a third runtime install and a dependency that can take the database offline. - Presented a valid certificate for the wrong name and watched
POL-02allow the issuance while PostgreSQL refused the login, because they answer different questions. - Established the order of checks, and that revocation is enforced in the handshake where no authentication rule can reach it.
- Switched the application to certificate authentication and confirmed the identity from the database's own view.
- Revoked the application's certificate and watched the server refuse it: the first time in twelve chapters that revocation protected the business.
- Retired two passwords, two roles and a six-step procedure, and were clear that Chapter 02 was not wrong, its constraint was removed.
Q1. SCRAM never puts the password on the wire and stores only a verifier. So what was actually wrong with it?
Nothing was wrong with it as a password scheme; it is a good one. The problem is structural
rather than cryptographic: two parties have to agree on a value, so the credential exists in two
places and a compromise of either is a compromise of it. Every part of PROC-01 was an attempt
to make that survivable. A private key is held by one party, and the database can verify it
without ever having seen it.
Q2. POL-02 allowed ca01 to obtain a certificate for db01.lab.simurgh.example, and the
login was refused anyway. Which control was wrong?
Neither. POL-02 answers "may this caller request this name", and ca01 may, and has been able
to since Chapter 07. PostgreSQL answers "may this certificate be this role", which is a different
question about a different thing. An estate with only the first control would let any certificate
its own authority ever issued log in as anything, which is exactly the Chapter 09 §1 attack with
the database as the target.
Q3. Why is it significant that sslv3 alert certificate revoked comes from the server rather
than the client?
Because it means revocation is enforced by the transport, before pg_hba is consulted, so it
cannot be disabled by changing an authentication rule. Somebody who switches cert back to
scram-sha-256 has changed what happens after the certificate is accepted; they have not
stopped the certificate being checked. Controls that live below the layer people edit are
unusually durable.
Q4. PROC-01 took six steps and two roles. Certificate replacement took one command. Was
Chapter 02 wasted?
No, and this is the most useful thing in the chapter. PROC-01 is the correct answer to its
constraint: changing a value two parties must agree on, with no window where neither works. The
six steps were not accidental complexity, they were the shape of that problem. Removing the
shared value removed the problem, and the complexity went with it. Complexity is usually a
constraint wearing a costume, and the way to find out is to remove the constraint.
Q5. OT-006 is called renamed rather than closed. What would closing it require?
A credential that is fetched at run time, used, and discarded, rather than read at startup and
held until the process exits. That means something short-lived: a certificate valid for hours
rather than ninety days, or a database credential issued on demand with a lease. Both need a
component that can mint credentials on request and expire them, which this estate does not have.
The private key on HOST-01 today lives exactly as long as the password did.
Q6. SVC-02 now holds nothing. Should it be deleted?
No, for two reasons. It is the only component in this estate that authenticates a caller from
kernel-supplied peer credentials and records every decision, and that machinery is worth keeping.
And the next secret the estate acquires will need somewhere to live. What is true is that
OT-011, a single point of total compromise holding everything in plaintext, currently holds
nothing worth compromising, and a component with no consumers is a component nobody will notice
breaking.
Q7. db01 refuses every client if its CRL goes stale. Is checking revocation worth that?
It is a real trade and the answer is not automatic. Without it, a compromised application
certificate works until it expires, which is up to ninety days. With it, an unattended pipeline
failure takes the database offline for everyone, and Chapter 11 measured that nothing on db01
reports how close that is. The honest position is that the check is worth it and it moved a
security problem into an availability one, which is D-066's trade appearing for the third time.
Q8. The application's key is 0400 owned by ACC-03. What exactly does that protect against,
and what does it not?
It stops every other unprivileged process on HOST-01, including SVC-02 and reportsvc, from
reading the key and becoming paymentsvc at the database. It does not stop root, which reads
everything and always has. So the workload separation this chapter introduces is exactly as
strong as the file permissions on one machine, which is stronger than a shared password and
weaker than it sounds.
sudo docker ps -a --format '{{.Names}}\t{{.Status}}'
curl -s http://127.0.0.1:8080/credinfo; echo
sudo docker exec -u postgres db01 crl-status --crl /var/lib/postgresql/crl/crl.pem
sudo docker exec db01 su postgres -c \
"psql -tAc \"SELECT usename, client_dn FROM pg_stat_ssl JOIN pg_stat_activity \
USING (pid) WHERE usename IS NOT NULL\""Expected: five machines Up and rootca Exited; "auth_method": "certificate"; both lists
reported ok on db01; and a live connection from CN=paymentsvc.
Nothing in this lab knows a database password any more. There is no value to rotate, no store entry to fetch, and no role that could be compromised by learning a string.
And there is still exactly one durable secret on HOST-01, read at startup, held until the
process exits, protected by a file mode. That is the next chapter, and it is the one the whole
build has been walking toward.