Compose dev

[jbaptperez]

IMPORTANT: All Pull Requests should be connected to an issue, if you don't
have an issue, please start by creating an issue and link it to the PR.

Please provide enough information so that others can review your pull request:

• What existing problem does this PR solve?
  • Issue Adapt the development Compose set-up to allow restricted company contexts #3225 (Adapt the development Compose set-up to allow restricted company contexts).
• What new feature is being introduced with this PR?
  • Add to the development set-up the possibility to customise image builds (among other) using a .env file,
  • Update of the Compose file syntax to the Compose specification,
  • Other minor enhancements related to the development Compose set-up
• Overview of changes to existing functions if required.
  • The current behavior remain the same, but the enhancement of the development set-up leads to different paths to the Dockerfiles and simplified commands to manage the development container.

Checks

• All tests succeed.
• Unit tests added.
• e2e tests added.
• Documentation updated.

Closing issues

Closes #3225.

Changes

.gitattributes and .dockerignore

Adds .gitattributes file to handle line endings, in particular when developing on Windows and building Linux images.
Adds .dockerignore file to limit the context transferred when building Docker images.

New Compose set-up for development

• Located at contrib/docker/dev.
• Does NOT replace the docker/dev set-up for now.
• Uses a latest Compose syntax.
• Relies only on Docker Compose CLI, not both Docker Compose and Docker CLIs anymore.
• Allows image builds in enterprise context: Optional dependency source proxies can be set for Ubuntu, Python and Node.
• Uses optimized multi-stage builds to locally build images.
• Centralizes variables in an non-versioned .env file, with a .env.template model file.
• Uses a structured file hierarchy for Compose services.
• Uses per service environment files.
• Uses only named volumes for speed, portability and to store all data that are not configuration or source files (use docker compose down -v to reset).
• Splits the main service (container) into one per component (setup, celery-worker, gunicorn, frontend).
• Makes the frontend service compatible with the 3 frontends implementations (switch via an environment variable).
• Dramatically optimizes the setup stage (dedicated service) (shallow clones/fetches the Sigma repository, calls tsctl once to register all rules at once).
• Uses a single entry point for each Docker service.
• Makes each service run on PID 1 to properly receive Docker signals.**
• Optimizes service startup order and leverages health checks to make service wait for each other (PostgreSQL, Redis, Gunicorn). OpenSearch has also a health check but the healthy state is not strictly required for development purposes.
• Adds a dedicated README.md to quickly get started.

Limits

I used a copy of the timesketch.conf file to properly set variables required for a local setup.
This copy must be updated according to the modification mades to the data/timesketch.conf file.

However, if you decides to replace the current Compose development setup with the new (contrib) one, this can be solved the following way:

• Move contrib/docker/dev to docker/dev,
• Adapt content (relative paths and README.md commands),
• Declare an environment variable for all settings that can be customized in the (Compose) timesketch.conf file,
• Adapt and test using Compose,
• Move the Compose timesketch.conf and replace the data/ one,
• Adapt the Compose references to the new location (bind mounts),
• Find references that use this file (development or contribution scripts) and adapt them: just declare and use environment variables.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[jbaptperez]

As I had time to update the PR, I also optimized some things, like using a Python virtual environment in the Docker image.
There is a lot of another improvements to be done, but maybe in another PR?

I thought about:

• Adding an environment variable to allow a custom place for the configuration files (with a default value to be compatible), this is necessary for the next steps,
• Put every component into its own container,
• Using Compose profiles to start everything or only a set of services,
• Allow, with port mapping to localhost, using our IDE to start a process in true debug mode (gunicorn or flask, celery),
• Use Compose development features for an optimized synchronization with the current source code.

[jkppr]

Thanks for the suggested changes to the development setup @jbaptperez. I appreciate you taking the time to propose these updates to improve the developer experience, especially for those in restricted environments. The changes are quite extensive and touch many core parts of our development setup, so we'll need some time to thoroughly review and test them.

While flexibility is important, we aim to maintain a simple and easily reproducible development environment that works out-of-the-box on a typical Linux machine. We generally encourage developers to manage their own custom configurations for specific environments, such as those found in corporate settings. Therefore, we may incorporate some of your proposed changes while declining others to ensure we keep the core setup as straightforward as possible.

[jbaptperez]

@jkppr, actually, the apparent change is not so big: Everything run into a single container with docker compose exec commands instead of docker exec ones.

I paid special attention to keeping default settings so that the behaviour without changes is the same as before.

A big change reported by Git is the deletion of the recently added yarn.lock file as it freezes a package source repository URL, which is indeed an issue for the PR.

The rest is close to the previous behaviour, with a single real difference: I duplicated the timesketch.conf file so that the development Compose set-up have its own; This simplifies the configuration from a single .env file but this is a kind of duplication, which can be discussed.

However, I understand your message and I am open to suggestions.

[jaegeral]

/gemini review

[gemini-code-assist]

Code Review

This pull request significantly enhances the development environment by overhauling the Docker Compose setup. Key improvements include the adoption of the modern Compose specification, the introduction of a .env file for easier customization of builds, and the use of a Python virtual environment within the container. The changes make the development setup more flexible and robust, especially for environments with network restrictions.

My review focuses on ensuring the new setup is correct and follows best practices. I've found a critical issue in the docker-entrypoint.sh script that would prevent the timesketch container from starting correctly due to an incorrect virtual environment path. I've also identified a couple of high-severity issues related to build reproducibility: the use of a :latest tag for a base image and ignoring package lock files in .gitignore. Finally, I've suggested some improvements to the new README.md for clarity and consistency.

[jbaptperez]

I rebased my branch on top of master.
This led me to update my work according to the current progress:

• Switching from Ubuntu 22.04 to Ubuntu 24.04,
• Switching from Node 18 to Node 20 (mandatory),

Note the virtualenv became the standard way to run the application in-between, which is great.

I made the following changes:

• Using local images built on-the-fly: They have their own directory, environment file, Dockerfile and they include the recently added health checks (with some changes),
• Using long syntaxes in the Compose file, which helps to understand the code,
• Changing the name of services and images, to cleary make the difference with any official images (e.g. PostgreSQL image names cannot conflict with the one for development),
• Other minor changes.

However, I the work is not over. To do:

• Splitting the timesketch service into individual ones:
  • Adding the development user and the sigma rules (once),
  • Starting the Celery worker,
  • Running Gunicorn (backend with auto-reload, static compiled frontend, port 5000), depends on the Celery worker,
  • Running Vite (dynamic frontend with auto-reload, port 5001, depends of Gunicorn),
• A proper update of the README.md.

[jbaptperez]

Hi @jkppr,
I finalized my work (1 year!) and wrote details about the new features introduced in the PR description.

For information, I'm moving to a new squad, so I won't work on this project anymore.
However, I transferred details about this PR to @jeflagel that remains in the team.

Sorry if I made such a big PR but I ran out of time to properly extract every new features that I added on-the-fly.

In the future, if you agree to replace the current Compose setup with the new one of this PR (for now in the contrib directory), the required maintenance of the new timesketch.conf file could be completely removed (see the limits section in the PR description).

I hope you will merge this PR.

[sasirven]

Hi any news about this Feature @jaegeral?

[jaegeral]

Hey @sasirven, I know I indicated in #3226 (comment) if it would be an option to put it in contrib, but now looking at it how it plaid out, I am concerned that this will create a nightmare to maintain. Every change we make to e.g. the timesketch.conf would have to be done in two places, we added OpenTelemetry to the project in #3805. The once the apparent change is not so big became a 1000 LOC addition with a lot of stuff being duplicate to other places in the repo.

Is there maybe a more robust way to get what you want to achieve? E.g. a new folder in contrib with a README.md with the problem description and a shell script that when executed copies all the files needed in place?

It also goes way beyond the initial idea to this: #3225 looking at this lis:

Located at contrib/docker/dev.
Does NOT replace the docker/dev set-up for now.
Uses a latest Compose syntax.
Relies only on Docker Compose CLI, not both Docker Compose and Docker CLIs anymore.
Allows image builds in enterprise context: Optional dependency source proxies can be set for Ubuntu, Python and Node.
Uses optimized multi-stage builds to locally build images.
Centralizes variables in an non-versioned .env file, with a .env.template model file.
Uses a structured file hierarchy for Compose services.
Uses per service environment files.
Uses only named volumes for speed, portability and to store all data that are not configuration or source files (use docker compose down -v to reset).
Splits the main service (container) into one per component (setup, celery-worker, gunicorn, frontend).
Makes the frontend service compatible with the 3 frontends implementations (switch via an environment variable).
Dramatically optimizes the setup stage (dedicated service) (shallow clones/fetches the Sigma repository, calls tsctl once to register all rules at once).
Uses a single entry point for each Docker service.
Makes each service run on PID 1 to properly receive Docker signals.**
Optimizes service startup order and leverages health checks to make service wait for each other (PostgreSQL, Redis, Gunicorn). OpenSearch has also a health check but the healthy state is not strictly required for development purposes.
Adds a dedicated README.md to quickly get started.

Which in itself might be better suited smaller PRs (or issues in the first place to discuss the problem to be solved). E.g.: Uses only named volumes for speed, portability and to store all data that are not configuration or source files (use docker compose down -v to reset). how does that contribute to the initial problem? If it is adding value, it could be in the main docker/dev file, or else we are creating a fork of the docker/dev file that, as stated above, becomes a nightmare to maintain and troubleshoot, as we would have to ask everyone who has problems, which setup they are using.

[sasirven]

Hey @jaegeral, sorry for the delay.

I agree that duplication would be a nightmare for you to maintain.
After looking at this in more detail, here’s my proposed approach:

• Provide a small script or clear instructions in the README explaining that users should copy the data folder locally in the docker/dev and override only the files they need (for example, timesketch.conf to set personal values). We'll need to update the compose to properly mount this custom folder instead of the root data one to work.
• Add an environment variable to properly mount the CA certificates so it works in a company environment.
• Include a .env.example file with all customizable values (e.g., pip and node PPA, CA certificates, etc.).
• For the release Docker setup, add a variable to select the repository for the Timesketch image, in case a company needs to override it.

This should make it much easier to run the project locally in a corporate environment withtout the maintenance nightmare.

I also noticed an issue with OIDC (this should be tracked in a separate issue): it currently requires an https scheme for the redirect_url in google_auth.py. This should be configurable via an environment variable (or similar) to support local OIDC redirects.