Skip to content

About

Backend of EcoPart application

Resources

Stars

0 stars

Watchers

2 watching

Forks

Repository files navigation

ecopart_back

Tests CI codecov

Backend of the EcoPart application

This repository contains the backend codebase for the EcoPart application, built using TypeScript, NodeJS, Express, SQLite, and Jest for testing. The architecture follows a clean structure, and the API is designed based on REST principles.

Installation

To install the necessary dependencies, run the following command:

npm install

Running the Backend in Development Mode

To run the backend in development mode with automatic restarts on code changes, use the following command:

npm run dev:watch

Running Tests and Generating Coverage Report

To execute tests and generate a coverage report, use the following command:

npm run test

Publishing a New Version of the Application

To publish a new version, create and push a new tag for the application's code. Pushing a tag in the GitHub repository triggers a GitHub Action that builds the Docker image and publishes it on Docker Hub. The Docker image can be found here.

git tag -a vXX.XX.XX -m "version message"
git push --follow-tags

Each tag push publishes the image under two Docker Hub tags:

Docker tag Meaning
ecopart_back:vXX.XX.XX Immutable. The exact build for that git tag — never overwritten.
ecopart_back:latest Moving. Always the most recently published version; the test server tracks it.

Docker Hub also holds ecopart_back:buildcache: the build's layer cache, reused by the next tag build. It is not a runnable image; never deploy it.

Once the image is published, the same workflow (.github/workflows/public.yml) automatically redeploys the test server (see "Automatic deployment to the test server" below).

The git tag says nothing about where the version is deployed — it only identifies a version of the code. Deployment to preprod is a separate, deliberate promotion step (see "Deployment to the preprod server" below), so that only a version already validated on the test server ever reaches preprod.

Automatic deployment to the test server

Each v*.*.* tag push runs a deploy-test job that, once the new :latest image is on Docker Hub, restarts the backend on the test server. Because the server sits behind a firewall, the job runs on a self-hosted GitHub Actions runner installed on the test server itself — no inbound SSH from GitHub is required.

The test server runs both the frontend and the backend from a single, hand-maintained docker-compose.yml (see "Production Procedure" below). The deploy job restarts only the backend (api) service — the frontend (web) is never touched, and the compose file itself is never modified by CI.

One-time setup on the test server (done by whoever administers the machine):

  1. Install the runner at the organization level (not the repo level): GitHub organization → Settings → Actions → Runners → New runner. A single org-level runner is shared by both the backend and the frontend (ecopart_front) repos, so each repo's deploy-test job can use it. Give it the label ecopart-test, then install it as a service so it survives reboots:

    ./svc.sh install && ./svc.sh start

    A repository-level runner only serves the repo it is registered to, so it could not be shared with the frontend. Register it once on the organization instead.

  2. Docker access: the runner's OS user must be in the docker group, and the server must have Docker Compose v2 (the docker compose plugin, not the legacy docker-compose v1). The docker-compose-plugin apt package only exists in Docker's official apt repo; if Docker was installed from Ubuntu's docker.io package (as on the current server), drop the plugin binary in directly instead:

    sudo mkdir -p /usr/local/lib/docker/cli-plugins
    sudo curl -SL https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64 \
      -o /usr/local/lib/docker/cli-plugins/docker-compose
    sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
    docker compose version   # must print v2.x

    If the server was previously running Compose v1 (docker-compose), migrate the stack once after installing v2 — v1 and v2 name containers differently, so v2 won't reuse the v1 containers and would otherwise collide on the published ports:

    cd /ecotaxadev2/ecopart/new_ecopart
    docker-compose down      # remove the old v1 api + web containers (data_storage is a bind-mount, untouched)
    docker compose up -d     # recreate both services under v2
  3. Deployment directory (DEPLOY_DIR in the workflow, set to /ecotaxadev2/ecopart/new_ecopart): the directory that already holds the test server's docker-compose.yml, its .env, and the persisted data_storage/. Update DEPLOY_DIR if this ever moves.

The job then runs, from that directory:

docker compose pull api      # pull the freshly published backend image
docker compose up -d api     # recreate ONLY the backend container (frontend `web` stays up)

Database migrations are applied automatically when the backend container boots, so the new version is fully migrated after up -d.

Isolation. The job pulls and recreates only the api service and removes only the previous backend image (and only if it changed and is no longer in use). The frontend, the compose file, and every other image/container on the host are left untouched. No host-wide docker image prune is run.

Deployment to the preprod server

Not active yet — the workflow is in place, but it needs a self-hosted runner on the preprod server before it can do anything. See "One-time setup" below.

Preprod is never deployed by a tag push. It is promoted on demand, from the GitHub UI:

Actions → Deploy to preprod server → Run workflow → enter the version (e.g. v0.0.67).

That runs .github/workflows/preprod.yml, which:

  1. Checks the requested vXX.XX.XX image exists on Docker Hub, and fails loudly if not.
  2. Re-points the moving :preprod Docker tag at that version, using docker buildx imagetools create. This is a registry-side manifest copy — no rebuild, no pull, and the multi-arch (amd64 + arm64) manifest is preserved.
  3. Restarts only the api service on the preprod server, exactly like the test job does.

The preprod docker-compose.yml therefore pins image: 'ecotaxa/ecopart_back:preprod' and never has to be edited to change version — CI moves the tag underneath it. A concurrency group serialises promotions so two runs can't race.

Because :preprod is a moving tag, it doesn't show which version is live. Every run writes the promoted version to its GitHub job summary, and on the server docker compose images api gives the running digest.

Rolling back is the same operation with an older version number — no revert commit, no rebuild.

One-time setup on the preprod server:

  1. Register a self-hosted runner with the label ecopart-preprod, following the exact same procedure as the test server above (register it at the organization level so the frontend repo can share it, then ./svc.sh install && ./svc.sh start). Same Docker prerequisites: the runner's OS user in the docker group, and Docker Compose v2.

  2. Create the deployment directory with its hand-maintained docker-compose.yml, its .env (from empty.env) and its data_storage/, following "Production Procedure" below — but with the api service pinned to :preprod instead of :latest, and on ports that don't collide with the test stack if both live on the same host.

  3. DEPLOY_DIR in .github/workflows/preprod.yml points at that directory (/home/ecopart/ecopart_pre_prod). Update it if the stack ever moves. The job fails immediately, with an explicit message, if the directory holds no usable compose file or if the api service does not pin ecotaxa/ecopart_back:preprod — a compose file left on :latest would make the deploy a silent no-op.

  4. Seed the :preprod tag by running the workflow once with a recent version. Versioned image tags only exist for versions published after this change, so v0.0.66 and earlier are not promotable — only :latest was pushed for those.

Checking which version a server runs

The tag workflow bakes the git tag into the image (APP_VERSION). On any server:

curl <API_URL>/version                                # {"version":"v0.0.78"}
docker compose logs api | grep "EcoPart API version"  # printed at every start

The Swagger page (/api-docs) shows the same version. Preprod reports the tag it was promoted from, since promotion re-points :preprod without rebuilding. An image built locally, or a run outside Docker, reports dev.

Production Procedure

The test/production server runs the frontend and backend together from a single docker-compose.yml, both pulling published images from Docker Hub:

services:
  api:
    image: 'ecotaxa/ecopart_back:latest'
    env_file: .env
    volumes:
      - ./data_storage:/src/data_storage
    ports:
      - "5005:4000"
    restart: unless-stopped
  web:
    image: 'ecotaxa/ecopart_front:latest'
    ports:
      - "3000:3000"
    restart: unless-stopped

To set up or update the server manually:

  1. Prepare the environment: copy empty.env to the server (next to the compose file) and rename it to .env, then set the variables. data_storage/ persists the SQLite DB + files across redeploys.

  2. Run the stack:

    docker compose pull && docker compose up -d

Note: the root docker-compose.yml in this repository builds the backend from source and is for local development only. The server uses the image-based compose shown above.

On the preprod server the compose file is identical except that the backend pins the promoted tag rather than the newest build:

  api:
    image: 'ecotaxa/ecopart_back:preprod'   # moved by the promotion workflow, not by tag pushes

If preprod shares a host with the test stack, give it its own directory, its own data_storage/, and non-colliding published ports.

Mounting a remote folder into the import remote/ directory

The import browser (endpoint GET /file_system/import_folders) lists the sub-folders of <DATA_STORAGE_IMPORT>/remote/. To make a remote/network storage (NFS, another local disk, an FTP export, …) importable from EcoPart, expose it in that remote/ directory as a symbolic link, then make sure the container can actually follow the link. There are two independent layers to get right — the host and the container.

1. On the host — create the symlink

cd <DATA_STORAGE_IMPORT>/remote          # e.g. <deploy_dir>/data_storage/ecopart_data_to_import/remote
ln -s /path/to/local/storage    my_folder        # link name → real target on the host
ln -s /path/to/nfs/mount        my_archive

The link name is what appears in the import browser; the target is the real path on the host (a mount point, an NFS mount, another folder, …). Check the targets resolve and are readable:

ls -la                                      # shows "my_folder -> /path/to/local/storage"
readlink -f my_folder && ls -ld "$(readlink -f my_folder)"

2. In the container — bind-mount the link targets at the same absolute path

A symlink stores a path, not the data. The container resolves that path inside its own filesystem, so the link target must be mounted into the container at the exact same absolute path it points to on the host — otherwise the container sees a broken/empty link and the folder appears empty (or the endpoint returns []).

In the api service of your docker-compose.yml:

services:
  api:
    volumes:
      - ./data_storage:/src/data_storage
      # one entry per symlink target, SAME path on both sides, read-only for import sources
      - /path/to/local/storage:/path/to/local/storage:ro
      - /path/to/nfs/mount:/path/to/nfs/mount:ro

Bind the actual mount points, not a shared parent. If several targets are separate mount points under one directory (e.g. two NFS mounts under the same parent), mount each one individually. Bind-mounting only the shared parent would expose empty stub directories, because the child mounts are not propagated into that bind.

Recreate the container so the new volumes take effect, then verify through the symlink:

docker compose up -d --force-recreate api
docker exec -it <api-container> sh -c 'ls /src/data_storage/ecopart_data_to_import/remote/ftp/'

Gotchas

  • Use Docker from the official apt repo, not the Snap package. The Snap Docker runs under strict confinement and cannot bind-mount arbitrary host paths outside $HOME (e.g. system or NFS mount points); the folders show up empty in the container no matter what. Install docker-ce via apt for any server that mounts system/NFS paths.
  • Mount propagation. The Docker daemon has its own mount namespace. If a target filesystem was mounted on the host after the daemon started, the daemon may not see it and will bind an empty directory. Fix by refreshing the daemon's view (sudo systemctl restart docker) and recreating the container, or by making the host mounts shared (sudo mount --make-shared <target>) and requesting propagation: rshared on the bind.
  • Read-only + permissions. Import sources should be mounted :ro. Files may be owned by numeric UIDs with no matching name in the container — that is fine as long as they are world-readable (r--/r-x for "others"), which is all the import needs.
  • Security note. The import browser follows any symlink placed under remote/, wherever it points — the path-traversal guard is intentionally permissive so these links work. Only trusted operators/admins should be able to create links in that directory.

Link with EcoTaxa

OpenAPI Documentation

The API is documented using the OpenAPI 3.0 specification. Documentation is auto-generated from @openapi JSDoc annotations in the router files and YAML schema definitions.

Interactive Swagger UI is available at /api-docs when the server is running (disable it by setting ENABLE_SWAGGER=false in your .env). The raw JSON spec is served at /api-docs.json.

Generate openapi.json (static file at project root):

npm run openapi:generate

openapi:generate is also run automatically on every git push via a versioned pre-push hook (.githooks/pre-push). If the generated openapi.json changed, the push is blocked so you can commit the updated file first.

Validate the generated spec:

npm run openapi:validate

How it works:

  • Router annotations live in src/presentation/routers/*.ts as @openapi JSDoc blocks.
  • Reusable schemas are defined in src/presentation/openapi/schemas/*.yaml.
  • The base OpenAPI config is in src/presentation/openapi/swagger-definition.ts.
  • swagger-jsdoc merges all of the above at runtime (Swagger UI) or build time (tools/generateOpenApi.ts).

When you add or modify an endpoint, add/update the @openapi annotation above it. If the request/response shape changes, update the corresponding YAML schema file as well.

Link with EcoTaxa

You can configure GENERIC_ECOTAXA_ACCOUNT_EMAIL in the .env file. Applicative accounts are created with this email in every EcoTaxa instances. You should then loggin to ecopart user that uses the same email as GENERIC_ECOTAXA_ACCOUNT_EMAIL and loggin for every instances to the related applicative ecotaxa account. For now you should reconnect to theses accounts every 30 day. We are working on an EcoTaxa feature to ease this process trough a refresh token or longer token.

EcoTaxa tokens at rest

The EcoTaxa token of every linked account is stored in the database. With ECOTAXA_TOKEN_ENCRYPTION_KEY set, these tokens are stored encrypted (AES-256-GCM), so a copy of the database (backup, local copy) does not give working EcoTaxa access. Without the variable, tokens are stored in clear as before, and the startup log says so.

Enabling the encryption on a server

Do this once on each server (test, preprod, production), in the directory that holds its docker-compose.yml and .env (/ecotaxadev2/ecopart/new_ecopart for test, /home/ecopart/ecopart_pre_prod for preprod).

  1. Generate a key, one per server, on any machine:

    openssl rand -base64 32

    Keep it wherever the other secrets of that server are kept. Never commit it, and never reuse the key of another server.

  2. Add it to the server .env:

    ECOTAXA_TOKEN_ENCRYPTION_KEY = <the generated key>

    Doing this before the version that supports encryption is deployed is harmless: older versions ignore the variable.

  3. Recreate the backend container so that it reads the new .env (a plain restart keeps the old environment):

    docker compose up -d --force-recreate api
  4. Check the startup log:

    docker compose logs api | grep "EcoTaxa token"
    • On the first start with the key: N EcoTaxa token(s) saved in clear are now encrypted. Later starts print nothing.
    • ECOTAXA_TOKEN_ENCRYPTION_KEY is not set means the variable did not reach the container: check the .env and step 3.
    • cannot be decrypted with the current key means the key differs from the one the tokens were encrypted with (see below).
    • A malformed key stops the server at startup with ECOTAXA_TOKEN_ENCRYPTION_KEY must be 32 bytes encoded in base64.
  5. Optionally, check the database from the host (requires sqlite3, path valid when DBSOURCE_FOLDER keeps its default data_storage/sqlite_db/). The count must be 0:

    sqlite3 data_storage/sqlite_db/<DBSOURCE_NAME> "SELECT count(*) FROM ecotaxa_account WHERE ecotaxa_account_token NOT LIKE 'enc:v1:%';"

Local development needs no key: leave the variable empty.

Good to know
  • Changing or losing the key makes the stored tokens unreadable. EcoPart then treats them like expired tokens, and the startup log counts them. Each user logs out of the EcoTaxa account in the EcoTaxa accounts tab of their EcoPart profile, then logs in again. Do the same for the generic accounts, from the EcoPart user that owns them (see above).
  • Rolling back to a version older than this feature has the same effect, because the older code cannot read encrypted tokens.
  • Copying a database to another server or to a development machine leaves its tokens unreadable there, unless the source server's key is used. That is the intended protection: accounts log in to EcoTaxa again on the copy.
  • Deleting an EcoPart user also deletes their EcoTaxa accounts, except the generic account (GENERIC_ECOTAXA_ACCOUNT_EMAIL), which EcoPart itself uses.

Database

EcoPart uses SQLite as its database. The database file location is configured via the DBSOURCE_FOLDER and DBSOURCE_NAME environment variables in the .env file.

Database Migrations

The project includes a lightweight migration system to manage database schema changes over time. Migrations are stored as TypeScript files in src/data/migrations/ and are automatically applied on application startup before any data sources are initialized.

Every time the Docker container starts (or restarts), the application automatically detects and runs all pending migrations in alphabetical order. This means deploying a new image that contains new migration files is all that is needed — no manual migration step is required. Already applied migrations are skipped, so the process is safe to run repeatedly.

Each migration file exports a migration object with:

  • id – A unique identifier (e.g. 000_initial_schema)
  • up(db) – Applies the migration
  • down(db) – Reverts the migration

Applied migrations are tracked in a _migrations table inside the SQLite database.

Migration Logs

All migration activity (successes, errors, and warnings) is logged in two places:

  1. Console output — visible via docker logs <container> or docker compose logs api.
  2. Log file — written to <DATA_STORAGE_FOLDER>/migrations.log (inside the Docker volume).

Each log entry includes an ISO timestamp, log level, and a descriptive message:

[2026-02-13T14:30:00.123Z] [MIGRATION] [INFO] ════════════════════════════════════════════════════════════
[2026-02-13T14:30:00.124Z] [MIGRATION] [INFO] Migration run started — 2 pending out of 3 total migration(s).
[2026-02-13T14:30:00.130Z] [MIGRATION] [INFO]   ↑ Applying migration: 001_add_status_column
[2026-02-13T14:30:00.145Z] [MIGRATION] [INFO]     ✓ 001_add_status_column applied successfully.
[2026-02-13T14:30:00.150Z] [MIGRATION] [INFO]   ↑ Applying migration: 002_create_audit_table
[2026-02-13T14:30:00.162Z] [MIGRATION] [INFO]     ✓ 002_create_audit_table applied successfully.
[2026-02-13T14:30:00.163Z] [MIGRATION] [INFO] Migration run complete — 2 migration(s) applied: 001_add_status_column, 002_create_audit_table
[2026-02-13T14:30:00.163Z] [MIGRATION] [INFO] ════════════════════════════════════════════════════════════

If a migration fails, the error is logged and the application stops to prevent the database from being left in an inconsistent state:

[2026-02-13T14:30:00.150Z] [MIGRATION] [ERROR]     ✗ Migration 002_create_audit_table failed. | SQLITE_ERROR: ...

To check the migration log from the server:

# Via Docker
docker compose exec api cat /src/data_storage/migrations.log

# Or directly on the mounted volume
cat ./data_storage/migrations.log

Migration CLI

Four npm scripts are available to manage migrations:

Command Description
npm run migrate:create -- <name> Create the next numbered migration file
npm run migrate:up Run all pending migrations
npm run migrate:down Rollback the last applied migration
npm run migrate:status Show which migrations are applied or pending

Creating a New Migration

npm run migrate:create -- add_column_to_project

This generates the next numbered file, for example 029_add_column_to_project.ts, in src/data/migrations/ with up and down stubs ready to fill in.

Example Migration

import { Migration } from "./migration-manager";
import { SQLiteDatabaseWrapper } from "../interfaces/data-sources/database-wrapper";

function runSQL(db: SQLiteDatabaseWrapper, sql: string, params: any[] = []): Promise<void> {
    return new Promise((resolve, reject) => {
        db.run(sql, params, function (err) {
            if (err) reject(err);
            else resolve();
        });
    });
}

export const migration: Migration = {
    id: "029_add_column_to_project",

    async up(db: SQLiteDatabaseWrapper): Promise<void> {
        await runSQL(db, "ALTER TABLE project ADD COLUMN status TEXT DEFAULT 'active';");
    },

    async down(db: SQLiteDatabaseWrapper): Promise<void> {
        // SQLite has limited ALTER TABLE support — column removal
        // may require recreating the table.
    },
};

Important Notes

  • Each migration runs in a transaction, together with its record in _migrations: a migration that fails halfway leaves the database as it was, and the server does not start. Do not write BEGIN, COMMIT or PRAGMA foreign_keys in a migration.
  • A migration that rebuilds a table (create a copy, DROP, RENAME) declares foreignKeysOff: true. SQLite ignores PRAGMA foreign_keys inside a transaction, so the runner switches foreign keys off before it, checks that the migration created no foreign key violation, then switches them back. Without the flag, dropping a parent table would cascade-delete its child rows.
  • Migrations run in alphabetical order by filename, which is why filenames are prefixed with a three-digit number (000_ to 029_ today). Two branches that each add a migration may pick the same number: renumber one of them before merging.
  • The initial migration (000_initial_schema.ts) captures the full existing schema using CREATE TABLE IF NOT EXISTS, making it safe on both fresh and existing databases.
  • Migrations are executed inside the application startup flow (in src/main.ts), so deploying a new version with new migration files is enough — no manual step is required.
  • For production rollbacks, use npm run migrate:down from the server or connect via the CLI tool at tools/migrate.ts.

Glossary

Clean Architecture is a software design principle that emphasizes separation of concerns and modularity, aiming to create a flexible and maintainable codebase. It divides the application into layers, such as presentation, domain, and data, ensuring that dependencies flow inward, maintaining a clear boundary between the layers. This approach allows developers to change the implementation details of one layer without affecting the others, promoting code reusability and testability.

REST API (Representational State Transfer API) is a set of guidelines and principles for designing web services that communicate over HTTP. It follows a stateless client-server architecture where each resource is uniquely identified by a URL, and standard HTTP methods (GET, POST, PUT, DELETE) are used to perform CRUD (Create, Read, Update, Delete) operations on these resources. REST APIs are designed to be scalable, simple, and easily integrated with various platforms, making them a popular choice for building web services.

About

Backend of EcoPart application

Resources

Stars

0 stars

Watchers

2 watching

Forks

Releases

Packages

Used by

Contributors

Languages