feat: September monitoring and automation SDK

README.md

@@ -54,6 +54,15 @@ func check(ctx context.Context) error {
 
 Generated service constructors and request/response types remain the primary SDK surface; the helper does not hide authorization, validation, pagination, operations, or billing behavior.
 
+For the coordinated, unreleased September APIs, see [Observability](docs/OBSERVABILITY.md):
+scoped automation/GitHub identity, monitoring endpoints, Grafana, alerts, and
+timestamped webhook verification with overlapping rotation.
+
+Read-only and receiver examples are in [examples/monitoring](examples/monitoring/main.go)
+and [examples/webhookreceiver](examples/webhookreceiver/receiver.go). Compile/check
+them without making API requests with `go test ./metalhost ./examples/...`.
+Managed GitHub runners are not part of the workflow identity integration.
+
 ## Keep the API snapshot synchronized
 
 Install pinned generation tools once:

docs/API_SURFACE.md

@@ -7,14 +7,18 @@ The SDK's public Go surface has two layers:
 
 ## Generated services
 
-The released snapshot includes:
+This checkout's snapshot includes the services below. The September entries
+remain coordinated-release previews; this list is not a claim that every
+published SDK version or production endpoint exposes them.
 
 - audit
 - bare metal
 - catalog
 - compute and SSH keys
 - health
 - IAM
+- scoped automation credentials, project service accounts, and GitHub workload identity
+- monitoring metrics, alert rules, verified destinations, incidents, and delivery tests
 - network
 - operations
 - projects and organizations
@@ -67,3 +71,27 @@ Compute exposes whole-VM backup capture and restore, including captured disk
 manifests and consistency. Storage exposes single-disk backup CRUD, disk restore
 via `from_snapshot`, and automatic backup schedules with keep-N retention.
 See [Backups API](BACKUPS.md) for retry, billing, timing, and restore semantics.
+
+## September observability (unreleased)
+
+The new `AutomationService`, `MonitoringService`, and `AlertService` contracts
+require the coordinated September backend release with automation and monitoring
+enabled. Existing IAM keys and sessions are unchanged.
+
+Creation/rotation requests accept a client UUID `request_id`. Retain the exact
+request on an ambiguous retry. Credential retries return the same credential's
+current metadata with `secret_unavailable=true`; no secret is stored for replay.
+Revoke a recovered key whose original secret was lost before creating another.
+
+Rule, destination, and incident mutations also carry a stable request ID and
+optimistic revision where applicable. A successful save is not confirmation that
+the native evaluator has applied it; inspect desired/applied revision and health.
+Queued delivery is not provider acceptance, and provider acceptance is not proof
+that a recipient read a notification. Missing data is never healthy zero usage.
+
+Hosted PromQL and Prometheus scrape endpoints are ordinary authenticated HTTP
+interfaces, not Connect RPCs. Use a project-scoped `monitoring.read` credential;
+the server derives immutable tenant identity and does not trust tenant headers.
+
+See [Observability](OBSERVABILITY.md) for runnable examples and explicit CLI/API
+boundaries. Workflow identity does not provision GitHub runners.

docs/OBSERVABILITY.md

@@ -0,0 +1,199 @@
+# September observability (unreleased)
+
+These contracts require the coordinated September backend release, enabled
+automation/monitoring features and provisioned telemetry storage. A generated
+client or successful save does not prove the deployment has enabled a feature.
+The APIs preserve existing browser sessions, legacy keys and billing behavior.
+
+## Credentials and GitHub Actions
+
+Use `aes.iam.v1.AutomationService` for project service accounts, explicit
+capabilities, expiring keys and GitHub trusts. Shared automation should use a
+service account; personal keys are appropriate for individual local tooling.
+`monitoring.read` permits metric reads, not VM changes or alert management.
+Trust permissions are intersected with the service account's current grants.
+
+Retain a UUID `request_id` and the exact request through ambiguous creation or
+rotation failures. `secret_unavailable` means the request already committed;
+the API returns metadata, never the original secret. Inspect/revoke that key
+before creating another. Overlap rotation is explicit and bounded to 24 hours.
+Legacy IAM rotation retains its original immediate-revocation semantics.
+
+For keyless GitHub Actions, use the customer portal's signed workflow-discovery
+flow and approve the observed immutable repository/owner and workflow identity.
+Do not trust a repository name alone. The workflow needs `id-token: write`;
+assertions must match the deployment's configured audience and fixed GitHub
+issuer. Pull-request events are not permitted. The exchanged token lasts 15
+minutes, has no refresh token, and can only use its configured project/grants.
+The September CLI branch implements `auth github` and `monitoring`; older
+CLI releases do not. Alternatively, use the
+[complete HTTP workflow example](https://metalhost.net/docs/developers/guides/github-actions)
+from the coordinated docs release. Exchange through
+`AutomationService.ExchangeGitHubToken` with `trust_name` and `assertion`; use
+the returned `access_token` as Bearer, scoped to the returned `project_name`.
+Mask both tokens and keep them in one step, not a config file, `GITHUB_ENV` or
+artifact. Long jobs need a fresh assertion/exchange. The trust's exact workflow,
+branch, event and environment must match verification. This is not GitHub sign-in
+and does not provision runners. Managed runners are planned separately.
+
+## Metrics, Grafana and Prometheus
+
+`Config.MonitoringEndpoints(project)` validates the API origin and canonical
+project name and returns:
+
+- `Prometheus`: Grafana datasource base; append `/api/v1/query` or
+  `/api/v1/query_range` for direct PromQL requests.
+- `ScrapeTargets`: authenticated Prometheus HTTP service discovery.
+- `Metrics`: one scrape shard; prefer discovery for automatic project sizing.
+
+Use HTTPS with `Authorization: Bearer <project monitoring.read credential>`.
+Never supply `X-Scope-OrgID`; the gateway derives tenant identity from current
+authorization. Endpoint construction itself does not authorize access.
+
+Grafana uses a Prometheus datasource (Mimir 3.2.1 compatibility), 30-second
+minimum interval and 10-second query timeout. Store the key in
+`secureJsonData.httpHeaderValue1`, not dashboard JSON. Download the Metalhost VM
+dashboard from the portal's Monitoring → Integrations page. Data-source-managed
+rule administration is not exposed; create managed alerts through Metalhost's
+structured rule API. Customer Grafana-managed alerts query the read API and
+remain subject to its budgets.
+
+Prometheus should use authenticated HTTP discovery and scraping, a 60-second
+interval, 10-second timeout, and `honor_timestamps: true`. Both discovery and
+scraping need `authorization.credentials_file`. Keep the credential file mode
+0600. The exporter omits expired samples instead of refreshing old observations.
+
+The initial hosted limit is seven days, minimum 30-second query steps, 10,000
+points per series, a 16-KiB public expression cap and a 4-MiB response
+cap. Concurrency is per project per gateway replica: four in earlier previews,
+twelve in the follow-up backend implementation, also subject to a global budget.
+Narrow queries on 422; back off on 429; show an unavailable state on service
+failure. Empty, unsupported and stale metrics are not zero or proof of health.
+Use `MonitoringService` for typed metric descriptors, VM summaries and curated
+chart responses with per-family quality. Metric samples are never invoices.
+
+### Runnable read-only example
+
+The [monitoring example](../examples/monitoring/main.go) uses this checkout's
+generated client and explicitly installs `Config.RoundTripper`; `Config.Client`
+does not add Bearer authentication on its own. Set `METALHOST_ENDPOINT` to an
+enabled HTTPS API origin, `METALHOST_VM` to a full resource name, and load
+`METALHOST_API_KEY` from your secret manager with `monitoring.read`, then run:
+
+```sh
+go run ./examples/monitoring
+```
+
+It queries one hour of CPU/memory, prints quality and sample counts, and changes
+no resources. Seven-day queries need a coarser step (for example 1800 seconds),
+not 60 seconds. Use returned bounds/step and retain quality alongside values.
+List calls use `next_page_token`/`page_token`; one page is not a whole inventory.
+
+This snapshot includes `SetEnhancedMonitoringPaused`. Use `GetEnhancedMonitoring`
+to obtain the current `installation_id`, then send the full VM `name`, that ID,
+`paused: true`, and a fresh UUID `request_id`. To resume, send `paused: false`
+with a new request ID. Retry an uncertain operation with its original ID and
+unchanged body. Pause/resume requires `monitoring.write`, preserves the
+installation and retained history, and does not stop/start the VM or uninstall
+software. Revocation is permanent. The portal and verified collector archive
+remain the installation entry points.
+
+For example, with an authenticated generated `MonitoringServiceClient`:
+
+```go
+response, err := client.SetEnhancedMonitoringPaused(ctx, connect.NewRequest(
+    &monitoringv1.SetEnhancedMonitoringPausedRequest{
+        Name: vmName, InstallationId: installationID,
+        Paused: true, RequestId: requestID,
+    },
+))
+```
+
+Retain `requestID` before sending; inspect `response.Msg.Monitoring.Status` after
+success. Resume uses the same installation, not a second enrollment.
+
+Project administrators can list all project credentials with
+`ListCredentialsRequest{ProjectName: project, AllProjectCredentials: true}`.
+Do not combine that flag with `service_account`. It permits inspection/revocation
+under the backend's authorization rules, not rotation of another member's key.
+
+## Alerts and destinations
+
+`AlertService` manages structured rules, current/future VM selections, previews,
+incidents, verified destinations and explicit test deliveries. Rules reconcile
+asynchronously: check desired/applied version and evaluation health after save.
+Preview defaults to the current sample. Set `history_seconds` to 3600, 21600 or
+86400 for a bounded historical estimate on up to 50 matching VMs. Inspect
+`history_status` and each resource's evaluated/expected sample counts: missing
+retained running intent or fresh telemetry produces gaps, not healthy samples.
+The episode estimate applies the proposed threshold and sustained duration; it
+does not replay routing, delivery or predict future notifications.
+Enhanced templates accept only their allowlisted exact `dimensions`; obtain
+mount/service/GPU names from observations rather than constructing PromQL.
+Acknowledging or snoozing an incident does not resolve it. A missing metric or
+collector outage must not manufacture recovery.
+`OPEN` means recovery has not been confirmed, not that the retained incident
+value is live. `RECOVERED` requires recovery evidence; `RETIRED` records a changed
+rule/resource context rather than fabricated recovery. Rule saving is a separate
+mutation from preview; use stable UUIDs and expected versions.
+
+Select existing project-member or webhook rows as destinations. Email needs
+recipient verification; destination pause stops future sending without changing
+incident history. `TestAlertDestination` is an explicit external send with a
+stable request UUID; poll `GetDestinationTest`. The latest retained result is
+also returned by `ListAlertDestinations`. `SENT` means provider/endpoint
+acceptance, not that a person read it.
+Slack, Discord and Teams destinations take a write-only provider webhook URL
+in `SaveAlertDestinationRequest.webhook_url` at creation. Read it from a secret
+manager or private input file, never source control or logs. Omit it when editing
+an existing destination. Changing that endpoint requires a new destination. Check the
+deployment's channel availability before asking recipients to verify or test.
+Scoped automation is an RPC allowlist: `monitoring.write` is not blanket access
+to every AlertService method. In particular, verification, destination tests
+and incident acknowledgements use the authorized human dashboard flow in the
+current implementation.
+
+## Verify and rotate webhook signatures
+
+The payload is at-least-once. Deduplicate its stable event IDs durably. Grouped
+payloads contain at most 100 detailed events but enumerate every included event
+ID; follow incident IDs for the full state rather than assuming details are
+complete. The stable delivery ID is reused for identical group retries; a
+different attempt UUID is generated on each HTTP request.
+
+Use `metalhost.VerifyWebhook(headers, rawBody, secret, time.Now())` before
+decoding the JSON. Limit bodies to 1 MiB before reading them, reject invalid
+signatures and keep a trustworthy clock. The verifier authenticates
+`timestamp.deliveryID.attemptID.rawBody` with HMAC-SHA256 and checks a five-minute
+clock/replay window. It does not implement durable deduplication for you. It
+never falls back to the legacy untimestamped header.
+
+See the compilable [receiver example](../examples/webhookreceiver/receiver.go).
+Supply an `Inbox` implementation that atomically stores the verified raw body
+with a unique delivery ID before returning 2xx. An in-memory map is insufficient
+across restarts/replicas. Decode/process from that durable inbox; deduplicate
+individual event IDs as well when grouped deliveries overlap.
+
+Check examples without making API requests:
+
+```sh
+go test ./metalhost ./examples/...
+```
+
+`WebhooksService.RotateSubscriptionSecret` requires the current secret version
+and a stable request UUID. A successful original call returns one new secret;
+retry recovery returns metadata plus `secret_unavailable`. Select immediate
+rotation to invalidate both older keys if a secret was lost or compromised.
+
+With a nonzero overlap (maximum 24 hours), `X-Metalhost-Signature-V2` contains
+`t=<unix>,v2=<new HMAC>,v2=<old HMAC>`. Either secret can verify it during that
+window. The original `X-Metalhost-Signature: sha256=<HMAC of raw body>` header
+continues using the old key until overlap expires, then uses the new key.
+Upgrade receivers to V2 before rotating; a legacy receiver cannot validate the
+new key through the original header until overlap ends. A second overlapping
+rotation is rejected while the first is active; immediate rotation is always
+available with the current version. Expired keys never sign new attempts.
+
+Webhook endpoints must be public HTTPS; redirects/private-address destinations
+are rejected. Test receivers must be explicitly approved fixtures. Never log
+the signing secret, credential response or raw support/customer payload.

examples/monitoring/main.go

@@ -0,0 +1,49 @@
+// Read-only monitoring example; requires an explicitly enabled API environment.
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"net/http"
+	"os"
+	"time"
+
+	"connectrpc.com/connect"
+	monitoringv1 "github.com/AES-Services/metalhost-sdk/gen/go/aes/monitoring/v1"
+	"github.com/AES-Services/metalhost-sdk/gen/go/aes/monitoring/v1/monitoringv1connect"
+	"github.com/AES-Services/metalhost-sdk/metalhost"
+)
+
+func main() {
+	cfg := metalhost.Config{Endpoint: os.Getenv("METALHOST_ENDPOINT"), APIKey: os.Getenv("METALHOST_API_KEY")}
+	vm := os.Getenv("METALHOST_VM")
+	if cfg.Endpoint == "" || cfg.APIKey == "" || vm == "" {
+		log.Fatal("set METALHOST_ENDPOINT, METALHOST_API_KEY and the full METALHOST_VM resource name")
+	}
+	cfg.HTTPClient = &http.Client{
+		Transport: cfg.RoundTripper(http.DefaultTransport),
+		Timeout:   30 * time.Second,
+		CheckRedirect: func(_ *http.Request, _ []*http.Request) error {
+			return http.ErrUseLastResponse
+		},
+	}
+	client := monitoringv1connect.NewMonitoringServiceClient(cfg.Client(), cfg.BaseURL())
+	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+	defer cancel()
+	end := time.Now().Unix()
+	result, err := client.QueryVMMonitoring(ctx, connect.NewRequest(&monitoringv1.QueryVMMonitoringRequest{
+		Name: vm, MetricIds: []string{"cpu_utilization_pct", "memory_used_bytes"},
+		StartTimeUnix: end - 3600, EndTimeUnix: end, StepSeconds: 60,
+	}))
+	if err != nil {
+		log.Fatal(err)
+	}
+	fmt.Printf("bounds=%d..%d step=%ds\n", result.Msg.StartTimeUnix, result.Msg.EndTimeUnix, result.Msg.StepSeconds)
+	for _, quality := range result.Msg.Quality {
+		fmt.Printf("%s: %s observed=%d\n", quality.MetricId, quality.Status, quality.ObservedAtUnix)
+	}
+	for _, series := range result.Msg.Series {
+		fmt.Printf("%s dimensions=%v samples=%d\n", series.MetricId, series.Dimensions, len(series.Samples))
+	}
+}

examples/webhookreceiver/receiver.go

@@ -0,0 +1,50 @@
+// Package webhookreceiver demonstrates bounded V2 verification and durable handoff.
+// It is not a standalone server: supply a durable Inbox and serve behind HTTPS.
+package webhookreceiver
+
+import (
+	"context"
+	"io"
+	"net/http"
+	"time"
+
+	"github.com/AES-Services/metalhost-sdk/metalhost"
+)
+
+// Inbox must atomically persist the verified body with a unique DeliveryID.
+// Duplicate IDs are successful no-ops, not errors. A worker processes persisted
+// bodies and deduplicates individual event IDs where grouped payloads overlap.
+// Scope the inbox to this subscription; never use an unverified header as a key.
+type Inbox interface {
+	PutOnce(context.Context, string, []byte) error
+}
+
+func Handler(secret string, inbox Inbox) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != http.MethodPost {
+			w.Header().Set("Allow", http.MethodPost)
+			http.Error(w, "POST required", http.StatusMethodNotAllowed)
+			return
+		}
+		if inbox == nil {
+			http.Error(w, "receiver unavailable", http.StatusServiceUnavailable)
+			return
+		}
+		body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20))
+		if err != nil {
+			http.Error(w, "invalid or oversized body", http.StatusBadRequest)
+			return
+		}
+		identity, err := metalhost.VerifyWebhook(r.Header, body, secret, time.Now())
+		if err != nil {
+			http.Error(w, "invalid signature", http.StatusUnauthorized)
+			return
+		}
+		// Acknowledge only after durable storage. Do not log the raw body or secret.
+		if err := inbox.PutOnce(r.Context(), identity.DeliveryID, body); err != nil {
+			http.Error(w, "receiver unavailable", http.StatusServiceUnavailable)
+			return
+		}
+		w.WriteHeader(http.StatusAccepted)
+	})
+}