ewoks install: fresh env by default with the pip-venv package manager

doc/conf.py

@@ -24,6 +24,7 @@
     "nbsphinx",
     "nbsphinx_link",
     "sphinx_copybutton",
+    "sphinx_tabs.tabs",
 ]
 templates_path = ["_templates"]
 exclude_patterns = ["build", "**.ipynb_checkpoints"]

doc/howtoguides.rst

@@ -15,6 +15,7 @@ Getting started
     howtoguides/inspect_workflows
     howtoguides/gui
     howtoguides/python
+    howtoguides/requirements
     howtoguides/engines
 
 Automation

doc/howtoguides/requirements.rst

@@ -0,0 +1,105 @@
+.. _requirements:
+
+Reproduce a workflow environment
+================================
+
+A :term:`workflow` stores the python environment it was created in as the ``requirements``
+field of its ``graph`` field. ``ewoks install`` recreates that environment and
+``ewoks execute --env`` runs the :term:`workflow` in it. See
+:ref:`this tutorial <install_tutorial>` for a step-by-step example.
+
+What is stored
+--------------
+
+``ewoks convert`` and ``ewoks execute -o convert_destination=...`` store
+
+* ``python`` and ``system``: the python interpreter and the operating system.
+* ``distributions``: every installed python package with its version and, when it was not
+  installed from the python package index, the git commit or the archive it came from. Any
+  :term:`package manager` can recreate the environment from this list.
+* ``manager``: the :term:`package manager` that generated the requirements, with the content
+  of the files it needs to recreate the environment: ``requirements.txt`` for pip-venv.
+
+Use ``--exclude-requirements`` to store nothing.
+
+.. note::
+
+    A :term:`workflow` that stores a list of requirements instead of the structure above is
+    still supported: the list is parsed as a ``requirements.txt`` file.
+
+Select a package manager
+------------------------
+
+``ewoks convert`` uses the :term:`package manager` of the current python environment. It is
+detected from the environment variables and files that package managers leave behind, and
+from the tool that installed most of the packages. Use ``--package-manager-name`` to select
+one explicitly
+
+.. code-block:: bash
+
+    ewoks convert demo demo.json --test --package-manager-name pip-venv
+
+``ewoks install`` uses the :term:`package manager` that generated the requirements, unless it
+is not installed on the machine. It installs the stored files of that :term:`package manager`.
+When the requirements do not contain them, or installing them fails, it generates its own
+files from the ``distributions`` list and installs those instead. Both installations are
+confirmed separately unless ``--yes`` is provided
+
+.. code-block:: bash
+
+    ewoks install demo.json --yes --package-manager-name pip-venv
+
+``--package-manager-command`` provides the command that invokes the tool, for example when it
+is not on the ``PATH`` or when a faster implementation should be used
+
+.. code-block:: bash
+
+    ewoks install demo.json --yes --package-manager-name pip-venv \
+        --package-manager-command /path/to/python
+
+Choose the environment
+----------------------
+
+.. code-block:: bash
+
+    # a root directory of your choice instead of the one of the package manager
+    ewoks install demo.json --yes --env-root /tmp/envs
+
+    # a name of your choice instead of the workflow identifier
+    ewoks install demo.json --yes --env-name demo-env
+
+    # remove the environment when it already exists
+    ewoks install demo.json --yes --clean
+
+    # another python version than the one stored in the requirements
+    ewoks install demo.json --yes --python-version 3.12
+
+    # add ewoks itself when the requirements do not contain it
+    ewoks install demo.json --yes --with-ewoks
+
+    # install in the current python environment instead of creating one
+    ewoks install demo.json --yes --in-place
+
+The default name is the identifier of the :term:`workflow`, which is the ``id`` field of its
+``graph`` field. A :term:`workflow` without an identifier gets a name derived from its
+content.
+
+Without ``--env-root`` the :term:`package manager` decides where the environment goes. venv
+creates an environment wherever it is told to, so for pip-venv ewoks uses ``~/.ewoks/envs``.
+
+An environment that already exists is installed in, which adds the requirements to what is
+already there. Use ``--clean`` to remove it first. Only a directory that contains a python
+environment is removed.
+
+The environment of a :term:`workflow` is a directory: for pip-venv it is a virtual
+environment. ``ewoks execute --env`` takes that directory, not the python interpreter inside
+it.
+
+Limitations
+-----------
+
+* ``--python-version`` is a request: pip-venv can only use the version of the python
+  interpreter that creates the environment. A warning is emitted when the version cannot be
+  provided.
+* A package installed from a local directory cannot be recreated elsewhere. This is reported as
+  a warning when the requirements are generated.

doc/reference/cli.rst

@@ -18,9 +18,27 @@ ewoks install
 
     If no ``requirements`` field exist, **ewoks install** will try to extract requirements from the :term:`tasks <Task>` in the :term:`workflows <Workflow>` before installing them.
 
-    Unless ``--yes`` is provided, **ewoks install** will ask for confirmation before installing the packages.
+    **ewoks install** installs the files of the :term:`package manager` that generated the ``requirements``.
+    When the ``requirements`` do not contain those files, or installing them fails, the Python ``distributions`` they contain are installed instead.
 
-    By default, packages are installed in the current Python environment: if **ewoks install** is run in a virtual environment, the packages will be installed in this virtual environment.
+    Unless ``--yes`` is provided, **ewoks install** will ask for confirmation before each installation and before removing an existing environment.
+
+    By default, packages are installed in a new Python environment named after the workflow identifier, which is the ``id`` field of its ``graph`` field.
+    The environment is created where the :term:`package manager` creates named environments (``~/.ewoks/envs`` for package managers that do not have such a directory).
+    Use ``--env-root`` for another directory to create the environment in and ``--env-name`` for another name than the workflow identifier.
+    An environment that already exists is installed in, unless ``--clean`` is provided to remove it first.
+    The :term:`workflow` can be executed in that environment when it contains **ewoks** itself, which is the case when the ``requirements`` contain it.
+    Use ``--with-ewoks`` to add **ewoks** when the ``requirements`` do not contain it, without changing any of the versions they do contain:
+
+    .. code-block:: bash
+
+        ewoks install myworkflow.json --yes --with-ewoks
+        ewoks execute --env ~/.ewoks/envs/myworkflow myworkflow.json
+
+    Use ``--in-place`` to install in the current Python environment instead. This is not supported by all package managers.
+
+    The package manager that generated the ``requirements`` is used to reproduce the environment when it is available.
+    Provide ``--package-manager-name`` (and optionally ``--package-manager-command``) to use another one: the ``requirements`` always contain the list of installed Python distributions as a fallback.
 
 ewoks convert
 -------------

doc/reference/glossary.rst

@@ -28,5 +28,8 @@ Glossary
     Execution engine
         An execution engine is the underlying software used to execute the :term:`workflow`. :term:`Ewoks` supports multiple execution engines: pypushflow, orange, dask and the ewoks internal excution engine.
 
+    Package manager
+        A package manager creates python environments and installs packages in them. :term:`Ewoks` uses package managers to store the environment in which a :term:`workflow` was created and to recreate it: pip with venv.
+
     blissdata
         `Blissdata <https://bliss.gitlab-pages.esrf.fr/blissdata>`_ is an API for accessing data from BLISS in memory.

doc/tutorials.rst

@@ -8,6 +8,7 @@ Tutorials
     tutorials/hello_world
     tutorials/create_workflow
     tutorials/execute
+    tutorials/install
     tutorials/infrastructure
     tutorials/external
     tutorials/develop

doc/tutorials/execute.rst

@@ -13,6 +13,17 @@ The :ref:`python interface <execute_python>` for executing :term:`workflows <Wor
 
 The equivalent exists for the :ref:`command-line <cli>`.
 
+A :term:`workflow` can also be executed in a python environment created from the
+:term:`workflow` itself
+
+.. code-block:: bash
+
+    ewoks install /path/to/graph.json --yes
+    ewoks execute --env ~/.ewoks/envs/mygraph /path/to/graph.json
+
+See :ref:`this tutorial <install_tutorial>` and
+:ref:`this how-to guide <requirements>`.
+
 The :code:`engine=None` argument selects the default :term:`execution engine <Execution engine>`. Documentation on different :term:`execution engines <Execution engine>`:
 
 * `ewoksppf <https://ewoksppf.readthedocs.io/>`_ : execute cyclic :term:`workflows <Workflow>`

doc/tutorials/getting_started.rst

@@ -87,6 +87,16 @@ To inspect or modify a workflow, convert it to a JSON file:
 
    ewoks convert demo demo.json --test
 
+The JSON file also stores the python environment the :term:`workflow` needs, so it can be
+recreated later:
+
+.. code-block:: bash
+
+   ewoks install demo.json --yes
+   ewoks execute --env ~/.ewoks/envs/demo demo.json --outputs=all
+
+See the `Install and execute a workflow tutorial <./install.html>`_.
+
 Learn More
 ==========
 

doc/tutorials/install.rst

@@ -0,0 +1,150 @@
+.. _install_tutorial:
+
+Install and execute a workflow
+==============================
+
+A :term:`workflow` needs the python packages of its :term:`tasks <Task>` to be installed.
+:term:`Ewoks` can store the python environment in which a :term:`workflow` was created inside
+the :term:`workflow` itself and recreate that environment later, on another machine or at
+another time.
+
+There are two sides to this
+
+* the *producer* creates a :term:`workflow` and stores its requirements,
+* the *re-producer* receives the :term:`workflow`, recreates the environment and executes the
+  :term:`workflow` in it.
+
+Both sides use a :term:`package manager`. The walk-through below does both sides
+
+.. toctree::
+    :maxdepth: 1
+
+    install/pip_venv
+
+The producer and the re-producer do not need the same :term:`package manager`: the
+requirements contain the installed python packages, which any :term:`package manager` can
+install. See :ref:`this how-to guide <requirements>` to select a :term:`package manager`.
+
+Store the requirements
+----------------------
+
+``ewoks convert`` saves the packages installed in the current python environment as the
+``requirements`` of the destination :term:`workflow`. The ``requirements`` field of the
+``graph`` field looks like this
+
+.. code-block:: json
+
+    {
+        "python": {"version": "3.12.11", "implementation": "CPython", "...": "..."},
+        "system": {"system": "Linux", "machine": "x86_64", "...": "..."},
+        "distributions": [
+            {"name": "ewokscore", "version": "5.1.0", "installer": "pip"},
+            {"name": "networkx", "version": "3.4.2", "installer": "pip"}
+        ],
+        "manager": {
+            "name": "pip-venv",
+            "version": "25.0.1",
+            "files": {"requirements.txt": "ewokscore==5.1.0\nnetworkx==3.4.2\n"}
+        }
+    }
+
+* ``distributions`` are the installed python packages. Any :term:`package manager` can
+  recreate the environment from this list.
+* ``manager`` is the :term:`package manager` that generated the requirements together with
+  the files it needs to recreate the environment exactly, for example a lock file.
+
+Create the environment
+----------------------
+
+``ewoks install`` creates a python environment for the :term:`workflow`. It prints the python
+interpreter of the environment it created, the command to execute the :term:`workflow` in it
+and the command to remove it again
+
+.. code-block:: text
+
+    Installed requirements for demo.json
+      Python : ewoks_envs/demo/bin/python
+      Execute: ewoks execute --env ewoks_envs/demo demo.json
+      Remove : rm -rf ewoks_envs/demo
+
+Without ``--yes`` you are asked to confirm after the packages have been listed. The
+walk-through uses ``--env-root`` to create the environment in the working directory. Without
+it the environment is created where the :term:`package manager` creates named environments.
+
+Execute the workflow
+--------------------
+
+``--env`` executes the :term:`workflow` with the python interpreter of that environment
+instead of the current one. This works because the requirements contain ``ewoks`` itself: it
+was installed in the environment in which the :term:`workflow` was converted. When they do
+not, add ``--with-ewoks`` to ``ewoks install`` to install it without changing any of the
+versions coming from the :term:`workflow`.
+
+Remove the environment
+----------------------
+
+The environment is a normal directory, so removing it is enough. This is the command that
+``ewoks install`` prints when it creates the environment.
+
+Limitations and caveats
+-----------------------
+
+Recreating a workflow environment does not guarantee that the workflow can be executed.
+The requirements stored in a workflow describe the Python environment, but a workflow can
+also depend on system software, external resources, configuration, or files that are not
+captured by the requirements.
+
+The workflow cannot be installed
+++++++++++++++++++++++++++++++++
+
+Environment creation can fail when the Python packages listed in the requirements cannot
+be installed in the target environment. For example:
+
+* A Python package requires a system package, compiler, or other native build dependency
+  that is not available on the target machine.
+* A package contains compiled code that is not compatible with the target operating system,
+  CPU architecture, or Python version.
+* A required package is no longer available from the configured package indexes, or requires
+  access to a private package repository.
+* A package has dependencies that cannot be resolved together with the required package versions.
+* The Python version required by a package is not available on the target machine.
+* Installing a package requires network access, credentials, a license, or another resource
+  that is not available.
+* The workflow was created on an operating system or architecture that is different from the
+  target system and one or more packages are platform-specific.
+* A package depends on external system libraries or runtime components that are not provided
+  by the Python package itself.
+
+The stored requirements can therefore make the Python environment reproducible,
+but they cannot guarantee that the environment can be recreated on every machine.
+
+The workflow can be installed but cannot be executed
+++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+A successfully recreated Python environment only guarantees that the Python dependencies
+can be installed. Execution can still fail because a task depends on resources or configuration
+outside that environment. For example:
+
+* A task input points to a file that does not exist on the target machine.
+* A task expects a directory, executable, configuration file, or other resource that
+  is not available.
+* A task relies on an environment variable that is not defined in the target environment.
+* A task requires access to an external service, database, network resource, or hardware device
+  that is unavailable.
+* A task requires credentials, secrets, or configuration that are not stored in the workflow.
+* A task assumes a particular working directory or filesystem layout.
+* A task uses operating-system features or commands that are not available on the target system.
+* A task relies on data that was available when the workflow was created but has not been
+  transferred with the workflow.
+* A task dynamically imports or installs Python packages that are not declared in the
+  workflow requirements.
+* A task depends on a specific version or configuration of external software that is not captured
+  by the Python environment.
+* A task produces or consumes temporary files whose locations or permissions differ on
+  the target machine.
+* The workflow relies on non-deterministic external state, such as the current date, remote data,
+  or the state of an external service.
+
+In these cases, `ewoks install` can successfully recreate the Python environment, while `ewoks execute`
+can still fail. The requirements should therefore be considered a description of the Python environment,
+rather than a complete description of everything required to execute a workflow.