diff --git a/.gitignore b/.gitignore index 37834b7d..62f80364 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,6 @@ experiments/*/ # JetBrains IDE idea directories .idea/ + +# Documentation +docs/build/ diff --git a/README.md b/README.md index 3f5d62bf..f2706aaf 100644 --- a/README.md +++ b/README.md @@ -1,44 +1,6 @@ # Synergistic Software Platform for AI, Physics Simulations, and Experiments (Synapse) -Synapse is a modular framework to build components of digital twins. -It supports operators of machines and experiments with ML-assisted predictions, trained on a combination of continously measured and simulated data. -Synapse embraces emerging [integrated research infrastructures](https://www.nersc.gov/what-we-do/computing-for-science/integrated-research-infrastructure), deploying a user-facing cloud service, using HPC/cloud compute, and exchanging modular components via container registries. - -At the moment, Synapse uses the NERSC Spin (control, dashboard), NERSC Superfacility API (simulation submission & ML training on Perlmutter), and NERSC container registry. -Synapse is actively developed and broadened to an AI-accelerated, portable framework. - -## Overview - -Synapse is a software platform that enables physicists to couple experimental data, simulations, and machine learning (ML) models trained on experimental and simulation data. -As an example, the schematic below illustrates how Synapse is used at the Berkeley Lab Laser Accelerator Center (BELLA): - -![Synapse overview](synapse_overview.png) - -One of the main software components is the graphical user interface (GUI), which is deployed via [Spin](https://docs.nersc.gov/services/spin/) at NERSC. -The application requires access to various data and information sources, as described below. - -### Displaying ML predictions - -To display ML predictions, the application requires the following: - -- **Experiment configuration file**: A YAML file named `config.yaml` stored in the root directory of an experiment's repository that defines the input, output, and calibration variables. -- **Simulation and experimental data points**: Each data point consists of values for the scalar inputs and outputs defined in the experiment configuration file. -Data points are stored in a [MongoDB](https://www.mongodb.com/) database, where each experiment is represented by a separate collection. -Experimental and simulation data points are stored in the same collection and are distinguished by the `experimental_flag` attribute. -- **ML models**: Machine learning models that interpolate between data points and are stored in [MLflow](https://mlflow.org/). -- **Simulation movies** (optional): For certain experiments, users can click on simulation data points to visualize simulation movies. -The corresponding MP4 files are stored in the Perlmutter shared file system at `/global/cfs/cdirs/m558/superfacility/simulation_data`. -This directory is mounted on the container image running on Spin. - -### Launching ML training at NERSC - -ML models can be trained by launching jobs on Perlmutter from the GUI, through the [NERSC Superfacility API](https://docs.nersc.gov/services/sfapi/). The application requires the following: - -- **Superfacility API credential file**: Instructions on generating and uploading the credential file from the GUI are in [dashboard/README.md](dashboard/README.md). -- **Submission script**: The batch script [ml/training_pm.sbatch](ml/training_pm.sbatch) is copied into the container image pushed to the NERSC registry and deployed via Spin (see [dashboard.Dockerfile](dashboard.Dockerfile)). It serves as a template for Superfacility API job submission when users launch model training from the GUI. -- **Python scripts and configuration files**: These include [ml/train_model.py](ml/train_model.py), [ml/Neural_Net_Classes.py](ml/Neural_Net_Classes.py), and the experiment configuration file `config.yaml`. -They are copied into the container image pushed to the NERSC registry and deployed via Spin (see [dashboard.Dockerfile](dashboard.Dockerfile)). -When users launch model training from the GUI, these files are copied to the Perlmutter shared file system at `/global/cfs/cdirs/m558/superfacility/model_training/src/` for access by the Superfacility API job. The `config.yaml` file is automatically populated with the configuration values specified in the GUI before being copied to the shared file system. +Synapse is documented in `docs/`. Start with the [overview](docs/source/overview.md), then see [dashboard](docs/source/dashboard.md), [ML training](docs/source/ml-training.md), and [experiment configuration](docs/source/experiment-configuration.md). ## Copyright Notice and License Agreement diff --git a/dashboard/README.md b/dashboard/README.md index 870bb169..bb724ffa 100644 --- a/dashboard/README.md +++ b/dashboard/README.md @@ -1,199 +1,3 @@ -# Table of Contents -* [Overview](#Overview) -* [Run the Dashboard Locally](#Run-the-Dashboard-Locally) - * [Without Docker](#Without-Docker) - * [With Docker](#With-Docker) -* [Run the Dashboard at NERSC](#Run-the-Dashboard-at-NERSC) -* [Get the Superfacility API Credentials](#Get-the-Superfacility-API-Credentials) -* [For Maintainers](#For-Maintainers) - * [Generate the conda environment lock file](#Generate-the-conda-environment-lock-file) - * [Build and push the Docker container to NERSC](#Build-and-push-the-Docker-container-to-NERSC) -* [References](#References) +# Dashboard -# Overview - -The Synapse dashboard provides a web interface for working with data from experiments, simulations, and ML models. - -The dashboard can be run in two distinct ways: - -1. Locally on your computer. - -2. At NERSC through Spin. - -# Run the Dashboard Locally - -This section describes how to develop and use the dashboard locally. - -## Without Docker - -### Prepare the conda environment - -1. Move to the [dashboard/](./) directory. - -2. Activate the conda environment `base`: -```bash -conda activate base -``` - -3. Install `conda-lock` if not installed yet: -```bash -conda install -c conda-forge conda-lock -``` - -4. Create the conda environment `synapse-gui`: -```bash -conda-lock install --name synapse-gui environment-lock.yml -``` - -### Run the dashboard - -1. Create an SSH tunnel to access the MongoDB database at NERSC (in a separate terminal): - ```bash - ssh -L 27017:mongodb05.nersc.gov:27017 @dtn03.nersc.gov -N - ``` - -2. Move to the [dashboard/](./) directory. - -3. Set up the database settings (read-only) and the AmSC MLflow API key: - ```bash - export SF_DB_HOST='127.0.0.1' - export SF_DB_READONLY_PASSWORD='your_password_here' # Use SINGLE quotes around the password! - export AM_SC_API_KEY='your_amsc_api_key_here' # Required when MLflow tracking_uri is AmSC - ``` - -4. Activate the conda environment `synapse-gui`: - ```bash - conda activate synapse-gui - ``` - -5. Run the dashboard as a web application: - ```bash - python -u app.py --port 8080 - ``` - -## With Docker - -### Run the dashboard - -1. Create an SSH tunnel to access the MongoDB database at NERSC (in a separate terminal): - ```bash - ssh -L 27017:mongodb05.nersc.gov:27017 @dtn03.nersc.gov -N - ``` - -2. Move to the root directory of the repository. - -3. Build the Docker image as described [below](#build-the-docker-image). - -4. Run the Docker container: - ```bash - docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' -e AM_SC_API_KEY='your_amsc_api_key_here' synapse-gui - ``` - For debugging, you can enter the container without starting the app: - ```bash - docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' -e AM_SC_API_KEY='your_amsc_api_key_here' -it synapse-gui bash - ``` - Note that `-v /etc/localtime:/etc/localtime` is necessary to synchronize the time zone in the container with the host machine. - -# Run the Dashboard at NERSC - -Connect to the [dashboard](https://bellasuperfacility.lbl.gov/) deployed at NERSC through Spin and play around! -Remember that you need to upload valid Superfacility API credentials in order to launch simulations or train ML models directly from the dashboard. - -# Get the Superfacility API Credentials - -Following the instructions at [docs.nersc.gov/services/sfapi/authentication/#client](https://docs.nersc.gov/services/sfapi/authentication/#client): - -1. Log in to your profile page at [iris.nersc.gov/profile](https://iris.nersc.gov/profile). - -2. Click the icon with your username in the upper right of the profile page. - -3. Scroll down to the section "Superfacility API Clients" and click "New Client". - -4. Enter a client name (e.g., "Synapse"), choose `sf558` for the user, choose "Red" security level, and select either "Your IP" or "Spin" from the "IP Presets" menu, depending on whether the key will be used from a local computer or from Spin. - -5. Download the private key file (in pem format) and save it as `priv_key.pem` in the root directory of the dashboard. - Each time the dashboard is launched, it will automatically find the existing key file and load the corresponding credentials. - -6. Copy your client ID and add it on the first line of your private key file as described in the instructions at [nersc.github.io/sfapi_client/quickstart/#storing-keys-in-files](https://nersc.github.io/sfapi_client/quickstart/#storing-keys-in-files): - ``` - randmstrgz - -----BEGIN RSA PRIVATE KEY----- - ... - -----END RSA PRIVATE KEY----- - ``` - -7. Run `chmod 600 priv_key.pem` to change the permissions of your private key file to read/write only. - -# For Maintainers - -## Generate the conda environment lock file - -1. Move to the directory [dashboard/](.). - -2. Activate the conda environment `base`: - ```bash - conda activate base - ``` - -3. Install `conda-lock` if not installed yet: - ```bash - conda install -c conda-forge conda-lock - ``` - -4. Generate the conda environment lock file: - ```bash - conda-lock --file environment.yml --lockfile environment-lock.yml - ``` - -## Build and push the Docker container to NERSC - -> [!WARNING] -> Pushing a new Docker container affects the production dashboard deployed at NERSC through Spin. - -> [!TIP] -> Run this workflow automatically with the Python script [publish_container.py](../publish_container.py): -> ```bash -> python publish_container.py --gui -> ``` - -> [!TIP] -> Prune old, unused images periodically in order to free up space on your machine: -> ```bash -> docker system prune -a -> ``` - -### Build the Docker image - -1. Move to the root directory of the repository. - -2. Build the Docker image: - ```bash - docker build --platform linux/amd64 --output type=image,oci-mediatypes=true -t synapse-gui -f dashboard.Dockerfile . - ``` - -### Push the Docker container - -1. Move to the root directory of the repository. - -2. Login to the [NERSC registry](https://registry.nersc.gov): - ```bash - docker login registry.nersc.gov - # Username: your NERSC username - # Password: your NERSC password without 2FA - ``` - -3. Tag the Docker image: - ```bash - docker tag synapse-gui:latest registry.nersc.gov/m558/superfacility/synapse-gui:latest - docker tag synapse-gui:latest registry.nersc.gov/m558/superfacility/synapse-gui:$(date "+%y.%m") - ``` - -4. Push the Docker container: - ```bash - docker push -a registry.nersc.gov/m558/superfacility/synapse-gui - ``` - -# References - -* [Using NERSC's `registry.nersc.gov`](https://docs.nersc.gov/development/containers/registry/) -* [Superfacility API authentication](https://docs.nersc.gov/services/sfapi/authentication/#client) +The Synapse dashboard is documented in [docs/source/dashboard.md](../docs/source/dashboard.md). diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 00000000..d0c3cbf1 --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = source +BUILDDIR = build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/docs.yml b/docs/docs.yml new file mode 100644 index 00000000..fb6df6a6 --- /dev/null +++ b/docs/docs.yml @@ -0,0 +1,12 @@ +name: synapse-docs + +channels: + - conda-forge + - nodefaults + +dependencies: + - myst-parser + - sphinx + - sphinx-autobuild + - sphinx-book-theme + - sphinx-copybutton diff --git a/docs/make.bat b/docs/make.bat new file mode 100644 index 00000000..747ffb7b --- /dev/null +++ b/docs/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=source +set BUILDDIR=build + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.https://www.sphinx-doc.org/ + exit /b 1 +) + +if "%1" == "" goto help + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/docs/source/conf.py b/docs/source/conf.py new file mode 100644 index 00000000..27d73546 --- /dev/null +++ b/docs/source/conf.py @@ -0,0 +1,30 @@ +# Configuration file for the Sphinx documentation builder. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Project information ----------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information + +project = "Synapse" +copyright = "BSD-3-Clause-LBNL" +author = "Arjun Dhamrait, Andrea Diaz, Marco Garten, Axel Huebl, Revathi Jambunathan, Remi Lehe, Ethan Rodriguez, Olga Shapoval, Jean-Luc Vay, Edoardo Zoni" + +# -- General configuration --------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration + +extensions = ["myst_parser", "sphinx_copybutton"] +myst_heading_anchors = 2 + +templates_path = ["_templates"] +exclude_patterns = [] + +# -- Options for HTML output ------------------------------------------------- +# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output + +html_theme = "sphinx_book_theme" +html_theme_options = { + "show_navbar_depth": 1, + "max_navbar_depth": 1, +} +html_static_path = [] diff --git a/docs/source/dashboard.md b/docs/source/dashboard.md new file mode 100644 index 00000000..f01bca7e --- /dev/null +++ b/docs/source/dashboard.md @@ -0,0 +1,216 @@ +# Dashboard + +The Synapse dashboard provides a web interface for working with experiment data, simulation data, and ML models. + +The dashboard can be run in two distinct ways: + +1. Locally on your computer. + +2. At NERSC through Spin. + +The dashboard is a Trame application rooted in `dashboard/app.py`. +It discovers experiments from `experiments/synapse-*`, reads each experiment's `config.yaml`, connects to MongoDB, loads MLflow models, and builds the GUI used to inspect data and launch jobs. + +## Run the Dashboard Locally + +This section describes how to develop and use the dashboard locally. + +### Without Docker + +#### Prepare the conda environment + +1. Move to the `dashboard/` directory. + +2. Activate the conda environment `base`: +```bash +conda activate base +``` + +3. Install `conda-lock` if it is not already installed: +```bash +conda install -c conda-forge conda-lock +``` + +4. Create the conda environment `synapse-gui`: +```bash +conda-lock install --name synapse-gui environment-lock.yml +``` + +#### Run the dashboard + +1. Create an SSH tunnel to access the MongoDB database at NERSC (in a separate terminal): + ```bash + ssh -L 27017:mongodb05.nersc.gov:27017 @dtn03.nersc.gov -N + ``` + +2. Move to the `dashboard/` directory. + +3. Set up the database settings (read-only) and the AmSC MLflow API key: + ```bash + export SF_DB_HOST='127.0.0.1' + export SF_DB_READONLY_PASSWORD='your_password_here' # Use SINGLE quotes around the password! + export AM_SC_API_KEY='your_amsc_api_key_here' # Required when MLflow tracking_uri is AmSC + ``` + +4. Activate the conda environment `synapse-gui`: + ```bash + conda activate synapse-gui + ``` + +5. Run the dashboard as a web application: + ```bash + python -u app.py --port 8080 + ``` + +### With Docker + +#### Run the dashboard + +1. Create an SSH tunnel to access the MongoDB database at NERSC (in a separate terminal): + ```bash + ssh -L 27017:mongodb05.nersc.gov:27017 @dtn03.nersc.gov -N + ``` + +2. Move to the root directory of the repository. + +3. Build the Docker image as described [below](#build-the-docker-image). + +4. Run the Docker container: + ```bash + docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' -e AM_SC_API_KEY='your_amsc_api_key_here' synapse-gui + ``` + For debugging, you can enter the container without starting the app: + ```bash + docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' -e AM_SC_API_KEY='your_amsc_api_key_here' -it synapse-gui bash + ``` + Note that `-v /etc/localtime:/etc/localtime` is necessary to synchronize the time zone in the container with the host machine. + +## Run the Dashboard at NERSC + +Connect to the [dashboard](https://bellasuperfacility.lbl.gov/) deployed at NERSC through Spin and explore it. +You need to upload valid Superfacility API credentials before you can launch simulations or train ML models directly from the dashboard. + +## Get the Superfacility API Credentials + +Follow the instructions at [docs.nersc.gov/services/sfapi/authentication/#client](https://docs.nersc.gov/services/sfapi/authentication/#client): + +1. Log in to your profile page at [iris.nersc.gov/profile](https://iris.nersc.gov/profile). + +2. Click the icon with your username in the upper right of the profile page. + +3. Scroll down to the section "Superfacility API Clients" and click "New Client". + +4. Enter a client name (e.g., "Synapse"), choose `sf558` for the user, choose "Red" security level, and select either "Your IP" or "Spin" from the "IP Presets" menu, depending on whether the key will be used from a local computer or from Spin. + +5. Download the private key file in PEM format and save it as `priv_key.pem` in the root directory of the dashboard. + Each time the dashboard is launched, it will automatically find the existing key file and load the corresponding credentials. + +6. Copy your client ID and add it on the first line of your private key file as described in the instructions at [nersc.github.io/sfapi_client/quickstart/#storing-keys-in-files](https://nersc.github.io/sfapi_client/quickstart/#storing-keys-in-files): + ``` + randmstrgz + -----BEGIN RSA PRIVATE KEY----- + ... + -----END RSA PRIVATE KEY----- + ``` + +7. Run `chmod 600 priv_key.pem` to restrict your private key file to read/write access only. + +## Main Managers + +- `state_manager.py`: shared Trame server, state, controller, and startup defaults. +- `model_manager.py`: MLflow model lookup, download, evaluation, and model training launch. +- `parameters_manager.py`: input sliders, parameter bounds, and single-simulation launch. +- `outputs_manager.py`: displayed output selection. +- `optimization_manager.py`: model-based input optimization with SciPy. +- `calibration_manager.py`: simulation-to-experiment variable conversion. +- `sfapi_manager.py`: Superfacility API credential upload, Perlmutter status, and job monitoring. +- `error_manager.py`: user-visible error collection. +- `utils.py`: config loading, database access, date filters, and Plotly figures. + +## Views + +- `/`: experiment selection, plots, parameter controls, optimization, ML controls, calibration controls, and errors. +- `/hpc`: NERSC Superfacility API credential and Perlmutter status panel. +- `/chat`: embedded assistant route for experiment support; currently backed by [synapse-chat.lbl.gov](https://synapse-chat.lbl.gov/). + +## NERSC Credentials + +Simulation and ML training launches require a Superfacility API key file uploaded through the dashboard. +The file must be PEM-formatted and include the Superfacility API client ID as the first line, followed by the private key. + +## For Maintainers + +### Generate the conda environment lock file + +1. Move to the directory `dashboard/`. + +2. Activate the conda environment `base`: + ```bash + conda activate base + ``` + +3. Install `conda-lock` if not installed yet: + ```bash + conda install -c conda-forge conda-lock + ``` + +4. Generate the conda environment lock file: + ```bash + conda-lock --file environment.yml --lockfile environment-lock.yml + ``` + +### Build and push the Docker container to NERSC + +```{warning} +Pushing a new Docker container affects the production dashboard deployed through Spin at NERSC. +``` + +````{tip} +Run this workflow automatically with the Python script `publish_container.py`: +```bash +python publish_container.py --gui +``` +```` + +````{tip} +Prune old, unused images periodically to free up space on your machine: +```bash +docker system prune -a +``` +```` + +#### Build the Docker image + +1. Move to the root directory of the repository. + +2. Build the Docker image: + ```bash + docker build --platform linux/amd64 --output type=image,oci-mediatypes=true -t synapse-gui -f dashboard.Dockerfile . + ``` + +#### Push the Docker container + +1. Move to the root directory of the repository. + +2. Log in to the [NERSC registry](https://registry.nersc.gov): + ```bash + docker login registry.nersc.gov + # Username: your NERSC username + # Password: your NERSC password without 2FA + ``` + +3. Tag the Docker image: + ```bash + docker tag synapse-gui:latest registry.nersc.gov/m558/superfacility/synapse-gui:latest + docker tag synapse-gui:latest registry.nersc.gov/m558/superfacility/synapse-gui:$(date "+%y.%m") + ``` + +4. Push the Docker container: + ```bash + docker push -a registry.nersc.gov/m558/superfacility/synapse-gui + ``` + +## References + +* [Using NERSC's `registry.nersc.gov`](https://docs.nersc.gov/development/containers/registry/) +* [Superfacility API authentication](https://docs.nersc.gov/services/sfapi/authentication/#client) diff --git a/docs/source/data-model.md b/docs/source/data-model.md new file mode 100644 index 00000000..4c2ae9aa --- /dev/null +++ b/docs/source/data-model.md @@ -0,0 +1,29 @@ +# Data Model + +MongoDB stores experiment and simulation records. +The collection name should match the `experiment` value in `config.yaml`. + +## Record Types + +- `experiment_flag: 1`: experimental data. +- `experiment_flag: 0`: simulation data. + +## Required Fields + +Each record must contain fields for the configured input and output variables, with names that match `config.yaml`. + +Simulation records may use simulation-space names when `simulation_calibration` maps those names back to experimental variables. + +## Optional Fields + +The dashboard uses these when present: + +- `date`: filtering and hover text for experimental records. +- `scan_number`: hover text. +- `shot_number`: hover text. +- `_id`: hover text and lookup for linked simulation media, such as MP4 files described in [Simulation Outputs](simulations.md#simulation-outputs). + +## Date Filtering + +Dashboard date filtering applies only to experimental records. +Simulation records are loaded without the date filter. diff --git a/docs/source/deployment.md b/docs/source/deployment.md new file mode 100644 index 00000000..257f69ae --- /dev/null +++ b/docs/source/deployment.md @@ -0,0 +1,34 @@ +# Deployment + +Synapse is deployed using Docker images and NERSC services. + +## Dashboard Image + +From the repository root: + +```bash +docker build --platform linux/amd64 --output type=image,oci-mediatypes=true -t synapse-gui -f dashboard.Dockerfile . +``` + +## ML Image + +From the repository root: + +```bash +docker build --platform linux/amd64 --output type=image,oci-mediatypes=true -t synapse-ml -f ml.Dockerfile . +``` + +The two build commands differ only by image tag and Dockerfile. + +## Publish Helper + +```bash +python publish_container.py --gui --ml +``` + +## NERSC Assumptions + +- Dashboard runs on Spin. +- Training and simulations run on Perlmutter through Superfacility API. +- Images are pushed to `registry.nersc.gov/m558/superfacility`. +- Before publishing, validate locally and, when possible, against a staging Spin deployment. diff --git a/docs/source/developer-notes.md b/docs/source/developer-notes.md new file mode 100644 index 00000000..61f916ac --- /dev/null +++ b/docs/source/developer-notes.md @@ -0,0 +1,52 @@ +# Developer Notes + +## Style + +Python code is linted and formatted with Ruff through pre-commit: + +```bash +pre-commit run --files +``` + +Ruff runs with its default rule set; there is no `pyproject.toml` or `ruff.toml` that overrides it. + +## Environments + +- Dashboard dependencies live in `dashboard/environment.yml`. +- ML dependencies live in `ml/environment.yml`. +- Regenerate the corresponding `environment-lock.yml` after dependency changes. + +## Documentation + +Create the documentation conda environment once from `docs/`: + +```bash +conda env create -f docs.yml +``` + +Build the documentation locally with: + +```bash +conda activate synapse-docs +cd docs +make html +``` + +The generated HTML is written to `docs/build/html/`. + +## Testing + +The project does not have a full pytest suite. +The main integration check is: + +```bash +python tests/test_ml_pipeline.py +``` + +It requires a local MLflow server. + +## Patterns + +- Dashboard features use manager classes in `dashboard/*_manager.py`. +- Experiment-specific behavior belongs under `experiments/synapse-*`. +- Shared dashboard helpers live in `dashboard/utils.py`. diff --git a/docs/source/experiment-configuration.md b/docs/source/experiment-configuration.md new file mode 100644 index 00000000..002e3f8a --- /dev/null +++ b/docs/source/experiment-configuration.md @@ -0,0 +1,59 @@ +# Experiment Configuration + +An experiment is a directory named `experiments/synapse-/`. +The dashboard strips `synapse-` and uses the rest as the experiment identifier. + +Clone the private repository for your experiment into the `experiments/` directory. + +Each experiment should provide: + +- `config.yaml` +- optional `simulation_scripts/` +- optional `experiment_scripts/` + +## Required Config Sections + +- `experiment`: collection and model namespace, for example `bella-ip2`. +- `database`: MongoDB connection and credential environment variables. +- `mlflow`: tracking URI and optional API key environment variable. +- `execution_mode`: ML training and simulation mode hints. +- `inputs`: scalar variables with `name`, `type`, `default`, and `value_range`. +- `outputs`: scalar variables with `name` and `type`. + +## Calibration + +`simulation_calibration` maps simulation variable names to experimental variable names: + +```yaml +simulation_calibration: + input1: + name: "simulation_variable" + unit: "unit" + depends_on: "experimental_variable" + alpha_guess: 1.0 + alpha_uncertainty: 0.1 + beta_guess: 0.0 + beta_uncertainty: 0.0 +``` + +Dashboard display uses: + +```text +experimental = simulation / alpha + beta +``` + +Simulation launch uses: + +```text +simulation = alpha * (experimental - beta) +``` + +These are inverse conversions: display maps simulation to experimental units, while launch maps dashboard parameters back to simulation units. + +## Add an Experiment + +1. Clone or create `experiments/synapse-/`. +2. Add `config.yaml`. +3. Ensure MongoDB fields match the configured input and output variable names. +4. Add `simulation_scripts/` only if dashboard launch is needed. +5. Train and register a model if dashboard predictions are needed. diff --git a/docs/source/getting-started.md b/docs/source/getting-started.md new file mode 100644 index 00000000..c08e41d6 --- /dev/null +++ b/docs/source/getting-started.md @@ -0,0 +1,40 @@ +# Getting Started + +For a reproducible installation, use `environment-lock.yml` rather than the unpinned `environment.yml`. + +## Dashboard + +From `dashboard/`: + +```bash +conda-lock install --name synapse-gui environment-lock.yml +conda activate synapse-gui +export SF_DB_HOST='127.0.0.1' +export SF_DB_READONLY_PASSWORD='...' +export AM_SC_API_KEY='...' +python -u app.py --port 8080 +``` + +For local MongoDB access, open a tunnel first: + +```bash +ssh -L 27017:mongodb05.nersc.gov:27017 @dtn03.nersc.gov -N +``` + +## ML Training + +From `ml/`: + +```bash +conda-lock install --name synapse-ml environment-lock.yml +conda activate synapse-ml +export SF_DB_READONLY_PASSWORD='...' +export AM_SC_API_KEY='...' +python train_model.py --test --config_file ../experiments/synapse-bella-ip2/config.yaml --model NN +``` + +## Required Environment Variables + +- `SF_DB_HOST`: MongoDB host for the dashboard. +- `SF_DB_READONLY_PASSWORD`: read-only MongoDB password. +- `AM_SC_API_KEY`: American Science Cloud MLflow API key when the config uses that service. diff --git a/docs/source/index.rst b/docs/source/index.rst new file mode 100644 index 00000000..8b24ced9 --- /dev/null +++ b/docs/source/index.rst @@ -0,0 +1,19 @@ +Synapse +======= + +Welcome to the Synapse documentation! + +.. toctree:: + :hidden: + :maxdepth: 1 + :caption: Contents: + + overview + getting-started + dashboard + data-model + experiment-configuration + simulations + ml-training + deployment + developer-notes diff --git a/docs/source/ml-training.md b/docs/source/ml-training.md new file mode 100644 index 00000000..c0ef2e32 --- /dev/null +++ b/docs/source/ml-training.md @@ -0,0 +1,302 @@ +# ML Training + +Synapse's ML training is implemented primarily in `ml/train_model.py`. + +ML models can be trained in two distinct ways: + +1. Locally on your computer. + +2. At NERSC, either manually or through the dashboard. + +It reads the configuration and MongoDB records, trains a model, wraps it with `lume-model`, and optionally registers it in MLflow. + +## Train ML Models Locally + +This section describes how to train ML models locally. + +### Without Docker + +#### Prepare the conda environment + +1. Move to the `ml/` directory. + +2. Activate the conda environment `base`: + ```bash + conda activate base + ``` + +3. Install `conda-lock` if it is not already installed: + ```bash + conda install -c conda-forge conda-lock + ``` + +4. Create the conda environment `synapse-ml`: + ```bash + conda-lock install --name synapse-ml environment-lock.yml + ``` + +#### Run the training + +1. Create an SSH tunnel to access the MongoDB database at NERSC (in a separate terminal): + ```bash + ssh -L 27017:mongodb05.nersc.gov:27017 @dtn03.nersc.gov -N + ``` + +2. Move to the `ml/` directory. + +3. Set up the database settings (read-only) and the AmSC MLflow API key: + ```bash + export SF_DB_READONLY_PASSWORD='your_password_here' # Use SINGLE quotes around the password! + export AM_SC_API_KEY='your_amsc_api_key_here' # Required when MLflow tracking_uri is AmSC + ``` + +4. Activate the conda environment `synapse-ml`: + ```bash + conda activate synapse-ml + ``` + +5. Run the ML training script in test mode: + ```bash + python train_model.py --test --model --config_file + ``` + +#### Test the full train/save/load cycle: `test_ml_pipeline.py` + +`tests/test_ml_pipeline.py` exercises the full ML lifecycle: training → upload to MLflow → download → accuracy check. It requires a local, empty MLflow server so it does not touch a production server. + +1. Start a local MLflow server, e.g. with Docker: + ```bash + docker run -p 127.0.0.1:5000:5000 ghcr.io/mlflow/mlflow mlflow server --host 0.0.0.0 + ``` + +2. Run the test script from the root of the repository (by default this expects the MLflow server to run on `localhost:5000`): + ```bash + python tests/test_ml_pipeline.py + ``` + + Optionally, restrict to a specific model type or config file: + ```bash + python tests/test_ml_pipeline.py --model NN --config_file experiments/synapse-bella-ip2/config.yaml + ``` + + If your MLflow server is running on a different port (e.g. 5001 instead of 5000), pass it explicitly: + ```bash + python tests/test_ml_pipeline.py --test-mlflow-uri http://localhost:5001 + ``` + +### With Docker + +Coming soon. + +## Train ML Models at NERSC + +This section describes how to train ML models at NERSC. + +### Manually without Docker + +#### Prepare the conda environment + +1. Move to the `ml/` directory. + +2. Activate your own user base conda environment: + ```bash + module load python + conda activate + ``` + +3. Install `conda-lock` if not installed yet: + ```bash + conda install -c conda-forge conda-lock + ``` + +4. Create the conda environment `synapse-ml`: + ```bash + conda-lock install --name synapse-ml environment-lock.yml + ``` + +#### Run the training + +1. Move to the `ml/` directory. + +2. Set up the database settings (read-only) and the AmSC MLflow API key: + ```bash + export SF_DB_READONLY_PASSWORD='your_password_here' # Use SINGLE quotes around the password! + export AM_SC_API_KEY='your_amsc_api_key_here' # Required when MLflow tracking_uri is AmSC + ``` + +3. Activate the conda environment `synapse-ml`: + ```bash + module load python + conda activate synapse-ml + ``` + +4. Run the ML training script in test mode: + ```bash + python train_model.py --test --model --config_file + ``` + +### Manually with Docker + +```{warning} +The Docker container is pulled from the [NERSC registry](https://registry.nersc.gov) and does not reflect any local changes you may have made to `train_model.py` unless you rebuild and redeploy the container first. +``` + +1. Log in to Perlmutter: + ```bash + ssh perlmutter-p1.nersc.gov + ``` + +2. Ensure the file `$HOME/db.profile` contains the read-only database password and the AmSC MLflow API key: `export SF_DB_READONLY_PASSWORD='your_password_here'` and `export AM_SC_API_KEY='your_amsc_api_key_here'`. + +3. Pull the Docker container: + ```bash + podman-hpc login --username $USER registry.nersc.gov + # Password: your NERSC password without 2FA + podman-hpc pull registry.nersc.gov/m558/superfacility/synapse-ml:latest + ``` + +4. Allocate a GPU node and run the container: + ```bash + salloc -N 1 --ntasks-per-node=1 -t 1:00:00 -q interactive -C gpu --gpu-bind=single:1 -c 32 -G 1 -A m558 + podman-hpc run --gpu -v /etc/localtime:/etc/localtime -v $HOME/db.profile:/root/db.profile -v /path/to/config.yaml:/app/ml/config.yaml --rm -it registry.nersc.gov/m558/superfacility/synapse-ml:latest python -u /app/ml/train_model.py --test --config_file /app/ml/config.yaml --model NN + ``` + Note that `-v /etc/localtime:/etc/localtime` is necessary to synchronize the time zone in the container with the host machine. + +### Through the dashboard + +````{warning} +When ML models are trained through the dashboard, Synapse uses NERSC's Superfacility API with the collaboration account `sf558`. +Because this is a non-interactive, non-user account, Synapse also uses a custom user to pull the image from the [NERSC registry](https://registry.nersc.gov) to Perlmutter. +The registry login credentials need to be prepared (only once) in the `$HOME` of user `sf558` (`/global/homes/s/sf558/`), in a file named `registry.profile` with the following content: +```bash +export REGISTRY_USER="robot\$m558+perlmutter-nersc-gov" +export REGISTRY_PASSWORD="..." +``` +```` + +Connect to the [dashboard](https://bellasuperfacility.lbl.gov/) deployed at NERSC through Spin and click the `Train` button in the `ML` panel. +You need to upload valid Superfacility API credentials before you can launch simulations or train ML models directly from the dashboard. + +## Model Types + +Use `--model` with one of: + +- `GP`: Gaussian Process. +- `NN`: single neural network. +- `ensemble_NN`: ensemble neural network. The current ensemble size is defined in `train_nn_ensemble()` in `ml/train_model.py`. + +## Command + +```bash +python train_model.py --config_file ../experiments/synapse-bella-ip2/config.yaml --model NN +``` + +Use `--test` to skip MLflow registration. + +## Phases + +1. Load config, variables, database records, and MLflow settings. +2. Build calibration and normalization transforms. +3. Train on simulation data. +4. Train the [calibration](experiment-configuration.md#calibration) on experimental data when available. +5. Build a `lume-model`. +6. Register to MLflow unless `--test` is set. + +## MLflow Names + +Registered models use: + +```text +synapse-_ +``` + +The MLflow experiment is: + +```text +synapse- +``` + +## For Maintainers + +### Generate the conda environment lock file + +1. Move to the directory `ml/`. + +2. Activate the conda environment `base`: + ```bash + conda activate base + ``` + +3. Install `conda-lock` if not installed yet: + ```bash + conda install -c conda-forge conda-lock + ``` + +4. Generate the conda environment lock file: + ```bash + conda-lock --file environment.yml --virtual-package-spec virtual-packages.yml --lockfile environment-lock.yml + ``` + +### Build and push the Docker container to NERSC + +```{warning} +Pushing a new Docker container affects ML training jobs launched from both locally deployed dashboards and the dashboard deployed at NERSC, because in both cases training runs in a Docker container pulled from the [NERSC registry](https://registry.nersc.gov). +Currently, this is the only way to test the end-to-end integration of the dashboard with the ML training workflow. +``` + +````{tip} +Run this workflow automatically with the Python script `publish_container.py`: +```bash +python publish_container.py --ml +``` +```` + +````{tip} +Prune old, unused images periodically to free up space on your machine: +```bash +docker system prune -a +``` +```` + +#### Build the Docker image + +````{important} +Ensure you have Docker version 29 or later [installed](https://docs.docker.com/engine/install/): +```bash +docker --version +``` +```` + +1. Move to the root directory of the repository. + +2. Build the Docker image: + ```bash + docker build --platform linux/amd64 --output type=image,oci-mediatypes=true -t synapse-ml -f ml.Dockerfile . + ``` + +#### Push the Docker container + +1. Move to the root directory of the repository. + +2. Log in to the [NERSC registry](https://registry.nersc.gov): + ```bash + docker login registry.nersc.gov + # Username: your NERSC username + # Password: your NERSC password without 2FA + ``` + +3. Tag the Docker image: + ```bash + docker tag synapse-ml:latest registry.nersc.gov/m558/superfacility/synapse-ml:latest + docker tag synapse-ml:latest registry.nersc.gov/m558/superfacility/synapse-ml:$(date "+%y.%m") + ``` + +4. Push the Docker container: + ```bash + docker push -a registry.nersc.gov/m558/superfacility/synapse-ml + ``` + +## References + +* [Using NERSC's `registry.nersc.gov`](https://docs.nersc.gov/development/containers/registry/) +* [Podman at NERSC](https://docs.nersc.gov/development/containers/podman-hpc/overview/) diff --git a/docs/source/overview.md b/docs/source/overview.md new file mode 100644 index 00000000..546c1907 --- /dev/null +++ b/docs/source/overview.md @@ -0,0 +1,92 @@ +# Overview + +Synapse is a modular framework for building digital twin components. +It supports machine and experiment operators with ML-assisted predictions trained on a combination of continuously measured and simulated data. +Synapse embraces emerging [integrated research infrastructures](https://www.nersc.gov/what-we-do/computing-for-science/integrated-research-infrastructure) by deploying a user-facing cloud service, using HPC/cloud compute, and exchanging modular components through container registries. + +At the moment, Synapse uses NERSC Spin (control and dashboard), the NERSC Superfacility API (simulation submission and ML training on Perlmutter), and the NERSC container registry. +Synapse is under active development and is being broadened into an AI-accelerated, portable framework. + +Synapse enables physicists to couple experimental data, simulations, and machine learning (ML) models trained on both experimental and simulation data. +As an example, the schematic below illustrates how Synapse is used at the Berkeley Lab Laser Accelerator Center (BELLA): + +![Synapse overview](synapse_overview.png) + +One of the main software components is the graphical user interface (GUI), which is deployed through [Spin](https://docs.nersc.gov/services/spin/) at NERSC. +The application requires access to various data and information sources, as described below. + +## Displaying ML Predictions + +To display ML predictions, the application requires the following: + +- **Experiment configuration file**: A YAML file named `config.yaml` stored in the root directory of an experiment's repository that defines the input, output, and calibration variables. +- **Simulation and experimental data points**: Each data point consists of values for the scalar inputs and outputs defined in the experiment configuration file. +Data points are stored in a [MongoDB](https://www.mongodb.com/) database, with each experiment represented by a separate collection. +Experimental and simulation data points are stored in the same collection and distinguished by the `experiment_flag` attribute. +- **ML models**: Machine learning models that interpolate between data points and are stored in [MLflow](https://mlflow.org/). +- **Simulation movies** (optional): For certain experiments, users can click on simulation data points to visualize simulation movies. +The corresponding MP4 files are stored in the Perlmutter shared file system at `/global/cfs/cdirs/m558/superfacility/simulation_data`. +This directory is mounted on the container image running on Spin. + +## Launching ML Training at NERSC + +ML models can be trained by launching jobs on Perlmutter from the GUI, through the [NERSC Superfacility API](https://docs.nersc.gov/services/sfapi/). The application requires the following: + +- **Superfacility API credential file**: Instructions on generating and uploading the credential file from the GUI are in [Dashboard](dashboard.md). +- **Submission script**: The batch script `ml/training_pm.sbatch` is copied into the container image pushed to the NERSC registry and deployed through Spin. It serves as a template for Superfacility API job submission when users launch model training from the GUI. +- **Python scripts and configuration files**: These include `ml/train_model.py`, `ml/Neural_Net_Classes.py`, and the experiment configuration file `config.yaml`. +They are copied into the container image pushed to the NERSC registry and deployed through Spin. +When users launch model training from the GUI, these files are copied to the Perlmutter shared file system at `/global/cfs/cdirs/m558/superfacility/model_training/src/` for access by the Superfacility API job. The `config.yaml` file is automatically populated with the configuration values specified in the GUI before being copied to the shared file system. + +## Workflow + +The main source areas are: + +- `dashboard/`: a Trame web application for exploring experiments, simulations, model predictions, optimization, calibration, and NERSC job controls. +- `ml/`: model training code for Gaussian Process, single Neural Network, and Neural Network ensemble models. +- `experiments/`: experiment-specific configuration and scripts, usually cloned from private repositories. +- `tests/`: integration checks for the ML pipeline. +- `docs/`: Sphinx documentation source. + +The typical workflow is: + +1. Add or update an experiment repository under `experiments/synapse-/`. +2. Define `config.yaml` with database, MLflow, input, output, and optional calibration settings. +3. Load experiment and simulation points from MongoDB. +4. Train a model from simulation data, optionally calibrating against experimental data. +5. Register the trained model in MLflow. +6. Use the dashboard to visualize data, query the model, optimize inputs, and launch NERSC jobs. + +## Services + +Synapse currently assumes these external services: + +- MongoDB for experiment and simulation records. +- MLflow for registered model storage. +- NERSC Spin for dashboard deployment. +- NERSC Superfacility API for Perlmutter jobs. +- NERSC container registry for dashboard and ML images. + +## Repository Map + +```text +dashboard/ Trame GUI and dashboard managers +ml/ ML training script, model classes, Perlmutter batch template +experiments/ Experiment configs and experiment-owned scripts +tests/ End-to-end ML pipeline helpers +docs/ Sphinx documentation source +``` + +## Copyright Notice and License Agreement + +Synapse v1.0 Copyright (c) 2025, The Regents of the University of California, +through Lawrence Berkeley National Laboratory (subject to receipt of +any required approvals from the U.S. Dept. of Energy). All rights reserved. + +If you have questions about your rights to use or distribute this software, +please contact Berkeley Lab's Intellectual Property Office at +IPO@lbl.gov. + +Please find the full copyright notice in `NOTICE.txt` and the full license agreement in `LICENSE.txt`. + +The SPDX license identifier is `BSD-3-Clause-LBNL`. diff --git a/docs/source/simulations.md b/docs/source/simulations.md new file mode 100644 index 00000000..52b605e4 --- /dev/null +++ b/docs/source/simulations.md @@ -0,0 +1,44 @@ +# Simulations + +Synapse treats simulation support as experiment-owned code. +The dashboard only needs to know where to find scripts and how to submit a job through the NERSC Superfacility API. + +## Directory Layout + +For dashboard-triggered single simulations, an experiment may provide: + +```text +experiments/synapse-/simulation_scripts/ + submission_script_single + templates/ + ... +``` + +If `submission_script_single` exists, the dashboard enables the `Simulate` button. +Before submission, it writes the current dashboard parameters to `single_simulation_parameters.yaml` after converting experimental variables to simulation variables. + +## Submission Flow + +1. User uploads valid Superfacility API credentials. +2. Dashboard checks Perlmutter status. +3. User clicks `Simulate`. +4. Files from `simulation_scripts/templates/` and the generated parameter YAML are uploaded to: + + ```text + /global/cfs/cdirs/m558/superfacility/simulation_running//templates + ``` + +5. The dashboard reads `submission_script_single` and submits it through Superfacility API. +6. Job status is polled until a terminal state, such as completed, failed, or cancelled. + +## Parameter Scans + +Some experiment repositories also include `submission_script_multi` or custom scan scripts. +These scripts are experiment-specific and are usually run manually on Perlmutter. + +## Simulation Outputs + +Simulation records should be written to the experiment's MongoDB collection with `experiment_flag: 0`. +Field names should match either the experiment config outputs or the configured simulation calibration variable names. + +Some dashboards may link simulation records to MP4 files stored on the Perlmutter shared file system, but this behavior is experiment-specific. diff --git a/synapse_overview.png b/docs/source/synapse_overview.png similarity index 100% rename from synapse_overview.png rename to docs/source/synapse_overview.png diff --git a/experiments/README.md b/experiments/README.md index 40edd62e..bbb8c01a 100644 --- a/experiments/README.md +++ b/experiments/README.md @@ -1,3 +1,3 @@ ## Experiments -Please clone the private repository relevant to your experiment in this directory. \ No newline at end of file +Please clone the private repository relevant to your experiment in this directory. See [docs/source/experiment-configuration.md](../docs/source/experiment-configuration.md) for the experiment configuration guide. diff --git a/ml/README.md b/ml/README.md index 2415deed..b4e9986d 100644 --- a/ml/README.md +++ b/ml/README.md @@ -1,269 +1,3 @@ -# Table of Contents -* [Overview](#Overview) -* [Train ML Models Locally](#Train-ML-Models-Locally) - * [Without Docker](#Without-Docker) - * [With Docker](#With-Docker) -* [Train ML Models at NERSC](#Train-ML-Models-at-NERSC) - * [Manually without Docker](#Manually-without-Docker) - * [Manually with Docker](#Manually-with-Docker) - * [Through the dashboard](#Through-the-dashboard) -* [For Maintainers](#For-Maintainers) - * [Generate the conda environment lock file](#Generate-the-conda-environment-lock-file) - * [Build and push the Docker container to NERSC](#Build-and-push-the-Docker-container-to-NERSC) -* [References](#References) +# ML Training -# Overview - -Synapse's ML training is implemented primarily in [train_model.py](train_model.py). - -ML models can be trained in two distinct ways: - -1. Locally on your computer. - -2. At NERSC, either manually or through the dashboard. - -# Train ML Models Locally - -This section describes how to train ML models locally. - -## Without Docker - -### Prepare the conda environment - -1. Move to the [ml/](./) directory. - -2. Activate the conda environment `base`: - ```bash - conda activate base - ``` - -3. Install `conda-lock` if not installed yet: - ```bash - conda install -c conda-forge conda-lock - ``` - -4. Create the conda environment `synapse-ml`: - ```bash - conda-lock install --name synapse-ml environment-lock.yml - ``` - -### Run the training - -1. Create an SSH tunnel to access the MongoDB database at NERSC (in a separate terminal): - ```bash - ssh -L 27017:mongodb05.nersc.gov:27017 @dtn03.nersc.gov -N - ``` - -2. Move to the [ml/](./) directory. - -3. Set up the database settings (read-only) and the AmSC MLflow API key: - ```bash - export SF_DB_READONLY_PASSWORD='your_password_here' # Use SINGLE quotes around the password! - export AM_SC_API_KEY='your_amsc_api_key_here' # Required when MLflow tracking_uri is AmSC - ``` - -4. Activate the conda environment `synapse-ml`: - ```bash - conda activate synapse-ml - ``` - -5. Run the ML training script in test mode: - ```bash - python train_model.py --test --model --config_file - ``` - -### Test the full train/save/load cycle: `test_ml_pipeline.py` - -[`tests/test_ml_pipeline.py`](../tests/test_ml_pipeline.py) exercises the full ML lifecycle: training → upload to MLflow → download → accuracy check. It requires a local, empty MLflow server to avoid touching any production server. - -1. Start a local MLflow server, e.g. with Docker: - ```bash - docker run -p 127.0.0.1:5000:5000 ghcr.io/mlflow/mlflow mlflow server --host 0.0.0.0 - ``` - -2. Run the test script from the root of the repository (by default this expects the MLflow server to run on `localhost:5000`): - ```bash - python tests/test_ml_pipeline.py - ``` - - Optionally, restrict to a specific model type or config file: - ```bash - python tests/test_ml_pipeline.py --model NN --config_file experiments/synapse-bella-ip2/config.yaml - ``` - - If your MLflow server is running on a different port (e.g. 5001 instead of 5000), pass it explicitly: - ```bash - python tests/test_ml_pipeline.py --test-mlflow-uri http://localhost:5001 - ``` - -## With Docker - -Coming soon. - -# Train ML Models at NERSC - -This section describes how to train ML models at NERSC. - -## Manually without Docker - -### Prepare the conda environment - -1. Move to the [ml/](./) directory. - -2. Activate your own user base conda environment: - ```bash - module load python - conda activate - ``` - -3. Install `conda-lock` if not installed yet: - ```bash - conda install -c conda-forge conda-lock - ``` - -4. Create the conda environment `synapse-ml`: - ```bash - conda-lock install --name synapse-ml environment-lock.yml - ``` - -### Run the training - -1. Move to the [ml/](./) directory. - -2. Set up the database settings (read-only) and the AmSC MLflow API key: - ```bash - export SF_DB_READONLY_PASSWORD='your_password_here' # Use SINGLE quotes around the password! - export AM_SC_API_KEY='your_amsc_api_key_here' # Required when MLflow tracking_uri is AmSC - ``` - -3. Activate the conda environment `synapse-ml`: - ```bash - module load python - conda activate synapse-ml - ``` - -4. Run the ML training script in test mode: - ```bash - python train_model.py --test --model --config_file - ``` - -## Manually with Docker - -> [!WARNING] -> Note that the Docker container is pulled from the [NERSC registry](https://registry.nersc.gov) and does not reflect any local changes you may have made to [train_model.py](train_model.py), unless you re-build and re-deploy the container first. - -1. Log in to Perlmutter: - ```bash - ssh perlmutter-p1.nersc.gov - ``` - -2. Ensure the file `$HOME/db.profile` contains the read-only password to the database and the AmSC MLflow API key: `export SF_DB_READONLY_PASSWORD='your_password_here'` and `export AM_SC_API_KEY='your_amsc_api_key_here'`. - -3. Pull the Docker container: - ```bash - podman-hpc login --username $USER registry.nersc.gov - # Password: your NERSC password without 2FA - podman-hpc pull registry.nersc.gov/m558/superfacility/synapse-ml:latest - ``` - -4. Allocate a GPU node and run the container: - ```bash - salloc -N 1 --ntasks-per-node=1 -t 1:00:00 -q interactive -C gpu --gpu-bind=single:1 -c 32 -G 1 -A m558 - podman-hpc run --gpu -v /etc/localtime:/etc/localtime -v $HOME/db.profile:/root/db.profile -v /path/to/config.yaml:/app/ml/config.yaml --rm -it registry.nersc.gov/m558/superfacility/synapse-ml:latest python -u /app/ml/train_model.py --test --config_file /app/ml/config.yaml --model NN - ``` - Note that `-v /etc/localtime:/etc/localtime` is necessary to synchronize the time zone in the container with the host machine. - -## Through the dashboard - -> [!WARNING] -> When we train ML models through the dashboard, we use NERSC's Superfacility API with the collaboration account `sf558`. -> Since this is a non-interactive, non-user account, we also use a custom user to pull the image from the [NERSC registry](https://registry.nersc.gov) to Perlmutter. -> The registry login credentials need to be prepared (only once) in the `$HOME` of user `sf558` (`/global/homes/s/sf558/`), in a file named `registry.profile` with the following content: -> ```bash -> export REGISTRY_USER="robot\$m558+perlmutter-nersc-gov" -> export REGISTRY_PASSWORD="..." -> ``` - -Connect to the [dashboard](https://bellasuperfacility.lbl.gov/) deployed at NERSC through Spin and click the `Train` button in the `ML` panel. -Remember that you need to upload valid Superfacility API credentials in order to launch simulations or train ML models directly from the dashboard. - -# For Maintainers - -## Generate the conda environment lock file - -1. Move to the directory [ml/](./). - -2. Activate the conda environment `base`: - ```bash - conda activate base - ``` - -3. Install `conda-lock` if not installed yet: - ```bash - conda install -c conda-forge conda-lock - ``` - -4. Generate the conda environment lock file: - ```bash - conda-lock --file environment.yml --virtual-package-spec virtual-packages.yml --lockfile environment-lock.yml - ``` - -## Build and push the Docker container to NERSC - -> [!WARNING] -> Pushing a new Docker container affects both the ML training jobs launched from a dashboard deployed locally and the ML training jobs launched from the dashboard deployed at NERSC, because in both cases the ML training runs in a Docker container pulled from the [NERSC registry](https://registry.nersc.gov). -> Currently, this is the only way to test the end-to-end integration of the dashboard with the ML training workflow. - -> [!TIP] -> Run this workflow automatically with the Python script [publish_container.py](../publish_container.py): -> ```bash -> python publish_container.py --ml -> ``` - -> [!TIP] -> Prune old, unused images periodically in order to free up space on your machine: -> ```bash -> docker system prune -a -> ``` - -### Build the Docker image - -> [!IMPORTANT] -> Ensure you have Docker version 29 or later [installed](https://docs.docker.com/engine/install/): -> ```bash -> docker --version -> ``` - -1. Move to the root directory of the repository. - -2. Build the Docker image: - ```bash - docker build --platform linux/amd64 --output type=image,oci-mediatypes=true -t synapse-ml -f ml.Dockerfile . - ``` - -### Push the Docker container - -1. Move to the root directory of the repository. - -2. Login to the [NERSC registry](https://registry.nersc.gov): - ```bash - docker login registry.nersc.gov - # Username: your NERSC username - # Password: your NERSC password without 2FA - ``` - -3. Tag the Docker image: - ```bash - docker tag synapse-ml:latest registry.nersc.gov/m558/superfacility/synapse-ml:latest - docker tag synapse-ml:latest registry.nersc.gov/m558/superfacility/synapse-ml:$(date "+%y.%m") - ``` - -4. Push the Docker container: - ```bash - docker push -a registry.nersc.gov/m558/superfacility/synapse-ml - ``` - -# References - -* [Using NERSC's `registry.nersc.gov`](https://docs.nersc.gov/development/containers/registry/) -* [Podman at NERSC](https://docs.nersc.gov/development/containers/podman-hpc/overview/) +Synapse ML training is documented in [docs/source/ml-training.md](../docs/source/ml-training.md).