Skip to content

fix(installation): add multi-cluster setup for ecs casting - #141

Open
Nageshbansal wants to merge 9 commits into
feat/runnerfrom
fix/casting/ecscasting
Open

fix(installation): add multi-cluster setup for ecs casting#141
Nageshbansal wants to merge 9 commits into
feat/runnerfrom
fix/casting/ecscasting

Conversation

@Nageshbansal

@Nageshbansal Nageshbansal commented Jun 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Features

  • Finds the cluster, subnets, security group and VPC by the substrate named in spec.infrastructure.name.
  • Runs ClickHouse and the keeper as multi-node clusters, each stateful node pinned to the instance holding its volume.
  • Works without a substrate too: state the identifiers as annotations and no lookups are emitted.
  • Pours backend.tf.json so the state location can be patched to a remote backend.
  • Adds provisioned/ and byo/ examples.

Example

Provisioned (an Infrastructure casting named foundry stamped the tags):

apiVersion: v1alpha1
kind: Installation
metadata:
  name: foundry
  annotations:
    foundry.signoz.io/ecs-region: us-east-1
spec:
  deployment:
    platform: ecs
    mode: ec2
    flavor: terraform
  infrastructure:
    name: foundry

Bring-your-own (every object stated, nothing looked up):

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
  annotations:
    foundry.signoz.io/ecs-region: us-east-1
    foundry.signoz.io/ecs-cluster-arn: arn:aws:ecs:us-east-1:111122223333:cluster/observability
    foundry.signoz.io/ecs-vpc-id: vpc-0a1b2c3d4e5f67890
    foundry.signoz.io/ecs-subnet-ids: subnet-0a1b2c3d4e5f67890,subnet-0f9e8d7c6b5a43210
    foundry.signoz.io/ecs-security-group-ids: sg-0a1b2c3d4e5f67890
spec:
  deployment:
    platform: ecs
    mode: ec2
    flavor: terraform

Fixes

  • AppConfig policy was scoped to the application instead of the configuration, so no component could read its config.
  • Migrator runs on Fargate and waits for its IAM policies before starting.
  • Refuses an unstated region and an empty id list at forge instead of failing in terraform.
  • A systemd lock no longer carries empty ECS annotations.

Refactors

  • One root module, one file per component, no child modules.

Tests

  • Covers identifier resolution and freezes the patch surface.

Chores

  • Rewrites the example README and corrects the annotation table.

Related: https://github.com/SigNoz/platform-pod/issues/2743

@Nageshbansal
Nageshbansal marked this pull request as ready for review July 22, 2026 11:07
@Nageshbansal Nageshbansal mentioned this pull request Aug 6, 2026
Merged
@Nageshbansal
Nageshbansal force-pushed the fix/casting/ecscasting branch from dca1c3e to 0bd60a6 Compare August 6, 2026 13:09
@Nageshbansal
Nageshbansal marked this pull request as draft August 6, 2026 13:12
@Nageshbansal
Nageshbansal force-pushed the fix/casting/ecscasting branch from 0bd60a6 to 0585dd5 Compare September 1, 2026 09:30
@Nageshbansal
Nageshbansal changed the base branch from main to feat/runner September 1, 2026 09:31
@Nageshbansal
Nageshbansal marked this pull request as ready for review September 1, 2026 09:34
@Nageshbansal Nageshbansal changed the title feat: add multi-cluster setup for ecs casting fix(installation): add multi-cluster setup for ecs casting Sep 2, 2026
@Nageshbansal
Nageshbansal force-pushed the feat/runner branch 2 times, most recently from 9e86648 to 52024fb Compare September 3, 2026 15:57
@Nageshbansal
feat(casting/ecs): find the substrate by tag and run multi-node clusters
390c931 
The casting placed tasks by naming every AWS object it needed. It now finds
its cluster, subnets and nodes by the tags the Infrastructure casting stamped
on them, so an installation states only which substrate it runs on and the
annotations remain for a cluster foundry did not provision.

Terraform runs through the terraform runner rather than the casting's own exec.

(cherry picked from commit 156012b6c8795ef87cc99ce60f6b255194e9b3a2)
@Nageshbansal
docs(reference): correct the ecs annotation table
b32e6e4 
Drops ecs-config-bucket, which no longer exists, fixes the task execution role
to map to execution_role_arn, and records that every annotation but the region
is an optional override of a tag lookup.
@Nageshbansal
fix(casting/systemd): stamp only the annotations of the mode being cast
8975266 
The enricher resolved every annotation in the installation catalog into the
lock, so a systemd casting advertised the ecs and eks annotations too, as nine
empty keys that mean nothing on a bare-metal host. Each entry carries the mode
it applies to; the loop now honours it.
@Nageshbansal
feat(casting/ecs): resolve every identifier on the casting, not in th…
a7d9447 
…e templates

Each of the cluster, vpc, subnets, security group and two roles arrives from one
of two sources: an annotation naming the object, or a derivation. That choice
was made inside the templates, three times over, by string-keyed lookups into
the annotation map, with the id lists split by template functions and nothing
validating what was stated.

A Reference now carries one axis resolved, and the casting decides. Templates
branch on the resolution rather than on a raw annotation, so no template names
an annotation key any more. Two mistakes that used to reach terraform are now
refused at forge: an unstated region, which rendered empty into tfvars and died
in terraform's own regex, and an id list that is set but names nothing.

Naming no infrastructure is the other shape. There is then no substrate to find
anything by, so the four axes it would derive have to be stated, and the pour
carries no lookup, no claim and no placement constraint. Stateful components
still run; nothing pins them to the instance holding their volume.

The schema migrator asked for FARGATE capacity in a cluster that registers EC2
container instances, so it could never place. It now runs on the cluster it
deploys to, beside the ephemeral pool, and depends on its task definition so
terraform defers the run to apply: the data source calls RunTask on every read,
and a plan is not the place to migrate a database.

The ingester was the only component the enricher never reached, so its OTLP
endpoints were missing from the lock while every other casting reported them.
@Nageshbansal
docs(examples/ecs): show the provisioned and the bring-your-own shape
8d1a5f6 
The example sat at the path gen-examples writes to, so every run overwrote the
hand-written casting with a bare one that this casting cannot forge. Both
examples now sit beside it, hand-written, and reproduce their own pours.

They differ only in whether the cluster is named: provisioned/ binds to a
substrate and its pour carries eight data sources, the claim chain and a pinned
placement for every stateful service; byo/ states each identifier and carries
none of the three.

The README claimed a module/ layout that no longer exists, config in S3 that is
AppConfig, a Fargate migrator that runs on EC2, and role name defaults derived
from the substrate rather than the casting. Its structure now follows the other
example READMEs.
@Nageshbansal
feat(casting/ecs): state the backend the pour already uses
b0b3c44 
Terraform writes state into the root module, which is the pour, and that is
where it belongs: the installation and the infrastructure are separate roots,
so melting one leaves the other's world alone. The casting never said so, and
that silence had a cost. A patch can only rewrite a material that exists, so
with no backend file there was no supported way to move state to S3 or GCS
short of editing a generated tree by hand.

backend.tf.json now states the local backend terraform would have defaulted to.
It changes no behaviour and exists to be patched over, the same file the
infrastructure casting pours for the same reason.
@Nageshbansal
fix(casting/ecs): scope the appconfig policy to what the action autho…
421a77d 
…rizes

StartConfigurationSession is authorized against the configuration, not the
application that holds it, and IAM matches an ARN literally: a grant on
application/X never covers application/X/environment/Y/configuration/Z. Every
component's agent was refused, so no component could read its own config.

GetLatestConfiguration still takes the whole account because it acts on a
session token and has no resource to name.
@Nageshbansal
fix(casting/ecs): run the migrator without waiting for a node to spare
a98d1ae 
A task in awsvpc mode takes an elastic network interface from the instance it
lands on, and an instance has few. The migrator is short-lived and owns no
data, so it runs on Fargate On-Demand and never competes for one with the
services that have to be somewhere. On-Demand needs no capacity provider
registered on the cluster; only Fargate Spot does.

The roles are named on the task definition, so they exist before the task
does, but what they are allowed to do arrives separately. A migrator that
starts before its execution role can pull an image fails for a reason that has
nothing to do with the migration, so the run waits on the policies too, and
only on the ones this casting creates.
@Nageshbansal
Nageshbansal force-pushed the fix/casting/ecscasting branch from 0585dd5 to a98d1ae Compare September 4, 2026 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@therealpandey therealpandey Awaiting requested review from therealpandey therealpandey is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Nageshbansal