Skip to content

development.md

Latest commit

 

History

History
396 lines (312 loc) · 16.6 KB

development.md

File metadata and controls

396 lines (312 loc) · 16.6 KB
Raw

Development

This page will help you get started developing on this application. We explain two ways:

  1. Using a devcontainer
  2. Classic local development

In a devcontainer {#devcontainer}

This is the recommended way if you want to get started simply, but it may not provide the full flexibility of the classic local development.

Prerequisites

Get started

  1. Open the cloned repository in VS Code.
  2. When prompted if you want to open the repository in a devcontainer, click "Reopen in Container".
  3. Wait for the devcontainer to build. This may take a few minutes.
  4. Once the devcontainer is ready, open a new terminal in VS Code.
  5. Run yarn start to in the VS code terminal to start backend and frontend.
  6. If it doesn't open automatically, open https://localhost:3000/ in your browser.

Classic local development {#classic}

Prerequisites

  1. Have NodeJS installed in the right version. Here is how to manage this:

    • Look for the key "engines" in /package.json to find out which versions are supported. The /.nvmrc file names the exact version the image and CI use.
    • You can use nvm to install, update, and switch NodeJS version in your system. As a homebrew user, use brew install nvm, then nvm list-remote to find available versions, nvm install <version> to install one and nvm use <version> to enable the version for the current shell session.

  2. Have yarn v1 installed for NodeJS dependency management. Use npm install --global yarn to install it.

  3. Credentials from 1Password:

    • GitHub OAuth credentials: You'll have to create a file /github-app-development-credentials.yaml in the clone repository, which for security reasons is not checked in with the repository. Find the content for this file in a 1Password secure note named Backstage Dev GitHub App.
    • Environment variables: Backstage requires a number of environment variables to be set in order to work successfully. Please see the section on Loading .env for details on how to create this file.

  4. A local configuration file named /app-config.local.yaml. Please copy /app-config.local.yaml.example for that purpose.

Database

docker run --rm -ti \
	--name backstage-postgres \
	-e POSTGRES_USER=backstage \
	-e POSTGRES_DB=backstage \
	-e POSTGRES_PASSWORD=secret-password \
	-e PGDATA=/var/lib/postgresql/data/pgdata \
	-v $PWD/volumes/postgres-data:/var/lib/postgresql/data \
  -p 5432:5432 \
	postgres:17.5

Tests on Postgres

A test declaring POSTGRES_18 in TestDatabases.create (for example packages/backend/src/searchEngine.test.ts) runs its Postgres cases only where BACKSTAGE_TEST_DATABASE_POSTGRES18_CONNECTION_STRING names a database; elsewhere it runs on sqlite alone. CI sets it to a Postgres 18 service beside the node-build job (.circleci/custom.yml), and under CI a missing variable fails the test instead of skipping. To run the Postgres cases locally:

docker run --rm -d --name backstage-test-postgres \
  -e POSTGRES_PASSWORD=postgres -p 55432:5432 postgres:18
BACKSTAGE_TEST_DATABASE_POSTGRES18_CONNECTION_STRING=postgresql://postgres:postgres@localhost:55432 \
  yarn test packages/backend/src/searchEngine.test.ts
docker stop backstage-test-postgres

Running the app locally

Loading .env

Backstage requires a number of variables that are defined in a .env file.

As many of the variables referenced are secrets, this file is not checked in by default and must be created.

Copy the file .env.example to .env and modify it to to replace the secret values. You can find the contents of the secret values in the 1Password secure note Backstage Dev Environment Variables.

Important

Only replace the values below the comment line referencing the secret. Do not replace or remove the variables above this point.

Once done, load the environment variables with:

set -a; source .env; set +a

(Simply running source .env will not work for sub-commands as the variables are not exported.)

If the environment variables are not loaded, yarn may fail to start with the following error:

Error: Failed to read config file at "/.../backstage/app-config.yaml", error at .integrations.github[0].apps[0], $include substitution value was undefined

Executing yarn

Make sure you have all the prerequisites mentioned above in place.

In the root directory of the cloned repository, execute

yarn install

before each attempt to run the app locally. This ensures that you have all dependencies installed in the right version.

To start both backend and frontent at the same time, execute

yarn start

Note that it can take a bit after launch until Backstage has processed the entire catalog data.

Once you make code changes, the development server will refresh the app automatically for you. No need to stop and start the server or hit the refresh button in your browser.

To stop the development server, hit Ctrl + C.

Fixture data

Some plugins can serve made-up data instead of calling their backends, set by a config flag: see Fixture modes. yarn fixtures start runs the portal with the Agent Platform's fixtures, signed in as the guest, on ports of its own: see Fixture mode.

Cluster access locally

Backstage reaches every management cluster through a single main Dex login configured as gs.authProvider.

Important

A main auth provider is required to run the app. The cluster-access status element in the sidebar resolves the main auth API (gsAuthApiRef) at startup, so an instance without gs.authProvider fails to render with No main auth provider configured. "gs.authProvider" configuration is missing. There is no "catalog-only / no auth" local mode -- pick one of the options below.

Two non-obvious values are needed before any login works locally; set them in your .env (see Loading .env) and reference them from app-config.local.yaml:

  • auth.session.secret: ${AUTH_SESSION_SECRET} -- the OIDC flow needs cookie session support; without it /api/auth/<provider>/start returns Authentication failed, authentication requires session support.
  • EXTERNAL_ACCESS_MCP_TOKEN -- referenced by the backend.auth.externalAccess block in app-config.local.yaml.example. It ships empty in .env.example; leaving it empty makes the backend fail to start with a "Missing required config value at backend.auth.externalAccess[0].options.token" error. Any non-empty string works for local dev.

Choose one of:

  • Single management cluster (simplest): use one installation's Dex as the main login and point the kubernetes plugin at that cluster. No broker is needed -- the main login token reaches the cluster directly. Mirror that installation's deployed config (in giantswarm-management-clusters/management-clusters/<mc>/extras/backstage/) into app-config.local.yaml, adapting the URLs to localhost:

    auth:
  environment: development
  session:
    secret: ${AUTH_SESSION_SECRET}
  providers:
    oidc-<mc>:
      development:
        metadataUrl: https://dex.<mc>.<baseDomain>/.well-known/openid-configuration
        clientId: ${AUTH_DEX_<MC>_CLIENT_ID}
        clientSecret: ${AUTH_DEX_<MC>_CLIENT_SECRET}
gs:
  authProvider: oidc-<mc>
  installations:
    <mc>:
      authProvider: oidc
      oidcTokenProvider: oidc-<mc>
      baseDomain: <mc>.<baseDomain>
      region: <region>
      pipeline: testing
      providers: [capa]
kubernetes:
  serviceLocatorMethod:
    type: multiTenant
  clusterLocatorMethods:
    - type: config
      clusters:
        - name: <mc>
          url: https://happaapi.<mc>.<baseDomain>
          authProvider: oidc
          oidcTokenProvider: oidc-<mc>

    The dev Dex client must allow the local callback https://localhost:7007/api/auth/oidc-<mc>/handler/frame. Use the Dex client that is registered for this localhost callback (the AUTH_DEX_<MC>_CLIENT_ID / _SECRET values in the 1Password note should point at it). Do not reuse a Dex client that is only registered for a deployed instance's callback URL -- a login attempt with it fails at Dex with Unregistered redirect_uri (often surfaced as a bare Error 400 if a gateway sits in front of Dex and masks the detail).

    If login instead fails at the token step with a 401 and the Dex log shows invalid client_secret on token request, the client secret in the 1Password note has drifted from the live Dex client; reconcile it before retrying.

  • Broker-backed multi-cluster access: point your instance at a muster cluster-token broker, which mints each cluster's Kubernetes token silently from your main session -- no per-cluster OIDC providers or login popups. In app-config.local.yaml set:

    • one main login provider under auth.providers (e.g. oidc-main) and gs.authProvider referencing it,
    • gs.clusterTokenBroker.tokenUrl (the muster /oauth/token endpoint) plus its confidential clientId / clientSecret,
    • each cluster under gs.installations.<mc> with clusterTokenAudience: <mc> and a matching kubernetes.clusterLocatorMethods entry.

    See /app-config.example.yaml for a complete, annotated example, and configuration.md for the field reference. The broker client id/secret and main Dex client credentials are the only cluster-access secrets you need; put them in your .env.

When the main session expires you get exactly one main-login prompt, then work resumes -- there are no per-cluster popups. Per-cluster access health (healthy / degraded / session-expired, with a reason) is shown by the cluster-access status element in the sidebar.

The legacy per-cluster "headless" backend (one backend + oidc-<mc> provider per cluster) has been retired; do not set up per-cluster providers or backendUrl overrides for local development.

Running backend and frontend separately

With the yarn start command above, two processes -- backend and frontend -- are logging into the same terminal. If this is not what you want, you can start both processes separately.

Execute these commands in separate shells (but watch out! Both need the environment variables set.)

  • yarn start backend to start the backend
  • yarn start app to start the frontend

Unit tests

yarn test runs jest through backstage-cli repo test once and exits. To run one test file, name it:

yarn test packages/backend/src/searchEngine.test.ts

Each change gets one such bounded run. yarn test:watch keeps jest running in watch mode and never exits: it is for a person at a terminal, never for a script, CI or an agent.

Backend wiring tests import features statically

A backend wiring test starts the features it proves with startTestBackend from @backstage/backend-test-utils, the way packages/backend/src/index.ts adds them to the real backend. The backend adds features with a dynamic import('…'); a test cannot: jest runs as CommonJS, and Node lets it load an ES module with import() only under --experimental-vm-modules, which backstage-cli repo test does not pass. A dynamic import() in a test fails with "A dynamic import callback was invoked without --experimental-vm-modules".

A new wiring test therefore:

  • imports each plugin and module it starts statically, as their default export (import searchPlugin from '@backstage/plugin-search-backend';);
  • passes them to startTestBackend({ features: [...] }), together with the mockServices factories it replaces (a database, config) and any test module of its own;
  • for a module alone, hands its extension points to startTestBackend({ extensionPoints: [...] }) instead of starting the plugin.

packages/backend/src/searchEngine.test.ts (a plugin with its engine module on real databases) and plugins/scaffolder-backend-module-gs/src/module.test.ts (a module against mocked extension points) are the examples.

End-to-end tests in a browser, against agentlab

e2e/agentlab/ is a Playwright suite that drives the portal the way a person does — the Dex sign-in popup, the Agent Platform tabs, the Connectors (MCP servers) page, the New agent wizard, a chat turn — against a running agentlab, the local kind cluster that runs the whole agent platform with this Backstage in it. It is the browser-level complement of agentlab backstage-test, which proves the same portal's routes and tokens headlessly and renders no page. The script first installs the Chromium its Playwright version needs, a no-op when it is there.

yarn test:e2e:agentlab                    # against https://backstage.127.0.0.1.nip.io
yarn test:e2e:agentlab --grep-invert lifecycle   # the pages only, ~1 min

The lab users (admin@lab.local, dev@lab.local, viewer@lab.local, password password) are the lab's throwaway fixtures, and the suite signs in as them. To test an unreleased build, put its image into the lab (platform.devImages.backstage in the lab's agentlab.yaml, then agentlab platform) and run the suite against it. A change to a page ships with the spec that proves it. Variables, coverage and the lab conditions that look like failures are in e2e/agentlab/README.md.

Manual UI test before merge

Automated tests cover the UI poorly, so a pull request that changes much of what people see is tested by hand before it merges. Its description has a ### UI impact section, written by its author with the ui-impact skill: a score from how many people see the change and how big it is, the pages it touches, and the steps to check it. A hidden block at the end of the section carries the same assessment for the triage that ranks pull requests, with the commit it was made at; a block made at an older commit than the head is stale and gets redone.

From score 6 the section starts with "Manual UI test needed before merge.", lists a test plan, and the pull request has the ui-test-needed label. It doesn't merge until someone has run the test plan against a build of the branch and replaced ui-test-needed with ui-tested. When a later push changes the UI again, ui-tested goes, and ui-test-needed comes back if the new score is still 6 or more.

This is a convention, not a gate: no check fails while ui-test-needed is set, and nothing notices a block that is behind the head. That happens whenever someone other than the author pushes (a co-author, the "Update branch" button, Renovate). Whoever merges checks the label, and that the block's commit is the head.

Running app locally with HTTPS

To use HTTPS with your local development site and access https://localhost, you need a TLS certificate signed by an entity your device and browser trust, called a trusted certificate authority (CA). The browser checks whether your development server's certificate is signed by a trusted CA before creating an HTTPS connection.

We recommend using mkcert, a cross-platform CA, to create and sign your certificate.

  1. Generate a certificate.

    [!IMPORTANT] On Linux, mkcert stores its CA under $CAROOT (default ~/.local/share/mkcert) but mkcert -install may need sudo to write the system trust store. Running the install under plain sudo uses root's CAROOT, creating a different CA than the one that signed your leaf certificate -- so the browser still distrusts https://localhost. Pin CAROOT for both steps, e.g. sudo env CAROOT="$HOME/.local/share/mkcert" "$(which mkcert)" -install, then generate the leaf cert with the same CAROOT.

  2. Put certificate data into certificate.yaml file in root of the project. See certificate.yaml.example for example.

  3. Add certificate configuration to app and backend packages. Use https instead of http in URLs:

app:
  baseUrl: https://localhost:3000
  https:
    $include: certificate.yaml
backend:
  baseUrl: https://localhost:7007
  cors:
    origin: https://localhost:3000
  https:
    $include: certificate.yaml