A small ecommerce API (Spring Boot + Postgres) instrumented with Micrometer, for learning Prometheus and Grafana on Kubernetes. A fake payment provider with tunable latency and failure rate lets you break things on purpose and watch the dashboards and alerts react.
app/ Spring Boot API (products, cart, checkout, fake payments)
k8s/ Kustomize base/ (app as a canary Rollout, Postgres, PodMonitor, SLO rules) + overlays/kind and k3s;
platform/ (Argo CD app-of-apps) and cluster/ (Gateway, routes, probe); monitoring values, k6 Job
terraform/ AWS: bootstrap (state, budget, CI role) and k3s on EC2 (Phase 4a)
prometheus/ Compose scrape config, SLO recording rules, burn-rate alerts, promtool tests
alertmanager/ routing, inhibition, Slack template, local webhook logger (secrets are gitignored)
scripts/ kind-up.sh / k3s-up.sh / argocd-up.sh (bootstrap; Argo CD does the rest), gen-k8s-rules.sh, experiments
loadtest/ k6 script: realistic browsing (default) or checkout-heavy stress
docs/ SLO spec, runbooks, postmortem template, roadmap, architecture notes
| Method | Path | Body |
|---|---|---|
| GET | /products | |
| GET | /products/{id} | |
| POST | /cart/items | {"cartId","productId","quantity"} |
| GET | /cart/{cartId} | |
| POST | /orders | {"cartId"} (201, 402, or 409) |
Free from Spring Boot: http_server_requests_seconds_* (tagged by uri, method, status),
JVM, and HikariCP pool metrics.
Custom (see StorefrontMetrics.java):
| Prometheus name | Type | Labels |
|---|---|---|
storefront_checkouts_total |
counter | status: success, payment_failed, out_of_stock |
storefront_cart_items_added_total |
counter | |
storefront_order_value_dollars_sum/_count |
summary | |
storefront_payment_duration_seconds_bucket |
histogram | outcome |
storefront_reservations_expired_total |
counter | (stock reservations released because checkout never finished) |
docker compose up --build
curl localhost:8080/products
curl -X POST localhost:8080/cart/items -H 'Content-Type: application/json' \
-d '{"cartId":"c1","productId":1,"quantity":2}'
curl -X POST localhost:8080/orders -H 'Content-Type: application/json' -d '{"cartId":"c1"}'
curl -s localhost:8080/actuator/prometheus | grep storefront_Read the raw /actuator/prometheus output before moving on. Knowing what a counter,
histogram bucket, and _sum/_count pair look like makes PromQL much easier.
Prometheus is on localhost:9090, and Grafana is on localhost:3000 (admin/admin) with the
Storefront > Storefront API and Storefront > Storefront SLOs dashboards already provisioned. On the SLO
dashboard, set Budget window to 1d in the lab, because a laptop never accumulates 28 days of data. If you edit
a dashboard JSON on Windows, run docker compose restart grafana: file changes don't reach the container
through the bind mount, so Grafana never sees them otherwise. Alertmanager is on localhost:9093. It routes
severity=page to Slack #alerts-page and severity=ticket to #alerts-ticket (setup:
alertmanager/secrets/README.md). It also sends every notification to a local
logger, so you can follow routing without Slack:
docker compose logs -f alert-loggerEach alert links to its runbook in docs/runbooks/. To put traffic on it:
docker compose --profile load run --rm k6The schema is managed by Flyway migrations in app/src/main/resources/db/migration, which also seed the catalog.
Hibernate only validates that the entities match (ddl-auto: validate). A database created before Flyway was added
has tables but no Flyway history, so the app refuses to start against it. In Compose, docker compose down removes
the old Postgres container (it has no volume), and the next up starts empty.
Tests run the app against a real Postgres through Testcontainers, so Docker must be running:
cd app && mvn verifyGitHub Actions runs four independent checks on every PR and push to main (also available through
Run workflow): Java 21 integration tests, Prometheus rule syntax/unit tests plus generated-rule drift,
Kubernetes schema validation/linting, and a two-minute Kubernetes SLO smoke test after warm-up.
Java test reports are saved as an artifact even on failures.
The workflow needs no cluster credentials or Slack secrets. Repository branch protection can require
Java integration tests, Prometheus rules, Kubernetes manifests, and Kubernetes SLO smoke test before merging.
To run the manifest check locally, install kubectl 1.34.1, kubeconform 0.8.0 and kube-linter 0.8.3 on your PATH:
bash scripts/ci-k8s-check.shThe check renders the base, kind, kind-gitops and CI overlays, and validates them plus Argo CD, the Gateway,
alert logger and k6 Job.
Custom-resource schemas come from a pinned CRD catalog revision; missing schemas fail the check.
Linting covers the deployable kind overlays, with documented exceptions in .kube-linter.yaml for CPU limits
and the existing PDB eviction policy. The supporting manifests receive schema validation only.
The Release workflow adds GHCR publishing, vulnerability gating, an SBOM, keyless signatures and Git image
promotion after successful main CI. Argo CD bootstrap/manifests are included for the registry-backed kind
overlay. First-release setup and the remaining GitOps/canary work are described in docs/delivery.md.
The SLO recording rules and burn-rate alerts (docs/slo.md) live in prometheus/rules/. Compose
loads them directly. After editing them, run the unit tests and regenerate the kind copies:
docker run --rm -v "$PWD/prometheus:/src" -w /src --entrypoint promtool prom/prometheus:v3.15.0 test rules tests/slo_test.yml tests/slo_alerts_test.yml
sh scripts/gen-k8s-rules.shOn Git Bash for Windows, prefix the docker run with MSYS_NO_PATHCONV=1 and use $(pwd -W) in place of $PWD.
Prerequisites: kind and kubectl (winget install Kubernetes.kind Kubernetes.kubectl on Windows), and about
9 GB of memory for Docker. On Docker Desktop with WSL 2 that's [wsl2] / memory=9GB in %USERPROFILE%\.wslconfig,
then wsl --shutdown and restart Docker Desktop. Stop the Compose stack first (docker compose down).
bash scripts/kind-up.shThe script is safe to rerun. It creates a 4-node cluster (k8s/kind-config.yaml: 1 control plane,
3 workers in fake zones a/b/c), installs Calico (kind's default network plugin doesn't enforce NetworkPolicy), loads
the Slack secret from alertmanager/secrets/, and installs Argo CD. Argo CD then installs everything else from
GitHub, in order: monitoring, the operators, the Gateway, and the app with its SLO rules and alerts
(k8s/platform, an app-of-apps). On a new cluster that takes 30-40 minutes, mostly pulling images.
So the cluster runs what's on main, not what's in your working tree. To try a branch, push it and run
REVISION=<branch> bash scripts/kind-up.sh. Alertmanager on kind uses the same alertmanager/alertmanager.yml as
Compose. Every command names the kind-storefront context, so other clusters in your kubeconfig are left alone.
Everything is reachable on port 80 through the cluster's Gateway (Envoy Gateway, k8s/gateway/). No port-forwards are
needed, and *.localhost resolves to 127.0.0.1 without any hosts-file entries:
| URL | What |
|---|---|
| http://storefront.localhost/products | the API (only /products, /cart, /orders; /actuator stays internal) |
| http://grafana.localhost | Grafana (admin/admin) |
| http://prometheus.localhost | Prometheus |
| http://alertmanager.localhost | Alertmanager |
Every notification also goes to the alert logger: kubectl --context kind-storefront -n monitoring logs -f deploy/alert-logger.
Put load on it:
kubectl --context kind-storefront -n storefront create configmap k6-script --from-file=loadtest/checkout.js
kubectl --context kind-storefront apply -f k8s/loadtest/k6-job.yamlThe storefront namespace enforces Pod Security restricted, and NetworkPolicies deny all traffic except the
paths the app needs. Check both directions with bash scripts/netpol-check.sh. A new pod or Job there needs a
restricted-compliant securityContext, and the app.kubernetes.io/component: loadtest label if it should reach the app.
In Prometheus, check Status > Targets for podMonitor/storefront/storefront-api. If it's missing,
the release: kps label on the PodMonitor is the first thing to check.
Three kind-on-Windows problems are already handled in the config. All were seen on this setup:
- etcd stalls on the virtual disk. Slow
fsyncthrough Docker Desktop's WSL disk made etcd time out, the API server drop out, and controllers crash-loop. etcd runs with--unsafe-no-fsync(lab only). - Postgres commits stall on the same disk. Under checkout load, an occasional seconds-long WAL flush pushed
checkout p95 to 2 s. The kind overlay sets
synchronous_commit: "off"(lab only), which brought p95 to about 240 ms. - Slow JVM startup. On a busy cluster the app took 60-70 s to start, and a fixed liveness delay killed it just
after it came up. A
startupProbenow gives it up to 3 minutes.
A finished version lives in grafana/dashboards/storefront.json, but building the panels yourself is
the fastest way to learn PromQL. Starter queries (paste into Grafana panels):
| Panel | PromQL |
|---|---|
| Requests/sec by endpoint | sum by (uri) (rate(http_server_requests_seconds_count{application="storefront-api"}[1m])) |
| p95 checkout latency | histogram_quantile(0.95, sum by (le) (rate(http_server_requests_seconds_bucket{uri="/orders",method="POST"}[5m]))) |
| Checkout outcomes/sec | sum by (status) (rate(storefront_checkouts_total[1m])) |
| Checkout failure ratio | sum(rate(storefront_checkouts_total{status!="success"}[5m])) / sum(rate(storefront_checkouts_total[5m])) |
| Revenue per minute | sum(rate(storefront_order_value_dollars_sum[5m])) * 60 |
| Average order value | sum(rate(storefront_order_value_dollars_sum[5m])) / sum(rate(storefront_order_value_dollars_count[5m])) |
| p95 payment latency | histogram_quantile(0.95, sum by (le) (rate(storefront_payment_duration_seconds_bucket[5m]))) |
| DB pool active / pending | hikaricp_connections_active and hikaricp_connections_pending |
| Pod restarts | kube_pod_container_status_restarts_total{namespace="storefront"} |
When your dashboard looks right, compare it with the committed one.
Each set env triggers a rollout. Predict what you'll see before running it.
- Payment outage.
kubectl -n storefront set env deploy/storefront-api PAYMENT_FAILURE_RATE=0.3The 5m burn rate jumps to about 60x right away, butStorefrontCheckoutAvailabilityBudgetBurn(severity page) waits until the 1h window passes 14.4x as well, about 15 minutes in. Set it back to 0.001 within half an hour and the page clears within about 5 minutes, because the 5m window recovers first. A longer outage also trips the 6h/30m pair, which takes up to 30 minutes to clear. Watch it on Prometheus's Alerts page. - Slow provider, two checkout designs.
kubectl -n storefront set env deploy/storefront-api PAYMENT_LATENCY_MS=1500Checkout p95 climbs with payment latency. Now repeat withCHECKOUT_MODE=single-tx, which calls payment inside the DB transaction. Each checkout keeps its product rows locked for the whole payment, so carts sharing a product queue behind each other, and checkout latency grows well past payment latency. The defaultsagamode reserves stock, commits, charges, then settles or returns the stock, so locks last milliseconds. Explaining the difference is a great interview answer. In Compose:CHECKOUT_MODE=single-tx docker compose up -d storefront-api. Then load it withdocker compose --profile load run --rm --no-deps -e PROFILE=checkout-stress -e VUS=30 k6. Without--no-deps,runrecreates the API with the default mode. Diagrams and trade-offs: docs/architecture/checkout.md. - Stockouts.
kubectl -n storefront set env deploy/storefront-api RESTOCK_LEVEL=20Theout_of_stockseries grows between restocks. - Scale out.
kubectl -n storefront scale deploy/storefront-api --replicas=3Watch per-pod series appear and notice why dashboard queries usesum by. - Kill a pod.
kubectl -n storefront delete pod -l app=storefront-api --wait=falseLook for the gap in the graphs and the scrape target going down.
Reset with PAYMENT_FAILURE_RATE=0.001 PAYMENT_LATENCY_MS=150 RESTOCK_LEVEL=200 CHECKOUT_MODE=saga.
make up builds the same system on three EC2 instances that you run yourself, and make down removes it. It
costs about $0.19/hr while up, and a nightly workflow destroys it in case you forget. Setup, cost and what
changes compared with kind: docs/k3s.md.
The original shortcuts (emptyDir Postgres, ddl-auto schema, a single replica, plaintext credentials in git) are fixed.
What's left is deliberate and limited to the lab, or not built yet.
Lab-only (kind or Compose, never in k8s/base/):
- etcd
--unsafe-no-fsyncand Postgressynchronous_commit: off, to work around Docker Desktop's slow virtual disk (k8s/kind-config.yaml,k8s/overlays/kind/). A crash of the Docker VM can lose the last moments of writes. - metrics-server
--kubelet-insecure-tls, because kind's kubelets use self-signed certificates. - Grafana
admin/admin, and Compose's fixed database credentials (storefront/storefront).
Not built yet:
- TLS on the Gateway (HTTP only; port 443 is mapped but unused).
- Database backups and a WAL archive. After an unclean restart, an instance that diverged had to be rebuilt instead of rewound (docs/ROADMAP.md 2.3).