diff --git a/CHANGELOG.md b/CHANGELOG.md index 130a07ba7afac..a7c082cdf5bdb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ ## Current develop ### Added (new features/APIs/variables/...) +- [[PR 1386]](https://github.com/parthenon-hpc-lab/parthenon/pull/1386) Add Python input file support (.py files) with argparse integration for programmatic parameter generation - [[PR 1382]](https://github.com/parthenon-hpc-lab/parthenon/pull/1382) Support Particle AMR - [[PR 1378]](https://github.com/parthenon-hpc-lab/parthenon/pull/1378) MeshData Swarm Tasks - [[PR 1377]](https://github.com/parthenon-hpc-lab/parthenon/pull/1377) Extend Initialization Hierarchy diff --git a/docs/python_input_summary.md b/docs/python_input_summary.md new file mode 100644 index 0000000000000..de48f14eb7b43 --- /dev/null +++ b/docs/python_input_summary.md @@ -0,0 +1,471 @@ +# Python Input Support - Implementation Summary + +## Overview + +This feature enables Python scripts as input files (`.py` instead of `.pin`), allowing programmatic parameter generation. The primary use case is native Python command-line argument parsing with full argparse support (choices, validation, help text, type conversion). The implementation is intentionally minimal: the core only provides a Python interpreter and bindings to `ParameterInput` methods. Everything else (helper classes, JSON parsers, etc.) is optional user-level tooling. + +## Key Capabilities + +**1. Python Command Line Arguments** +```bash +./app -i input.py --ndim=3 --nx=128 --problem=blast --cfl=0.3 --help +``` + +Python scripts can use argparse for configuration: +- Type checking: `--nx=128` (enforces int) +- Choices: `--problem` in {blast, linear_wave, kh} +- Help text: `--help` shows all available options +- Defaults: fallback values if not specified +- Custom validation: complex constraints on parameter combinations + +**2. Programmatic Configuration** +- Loops, conditionals, math expressions in parameter definitions +- Dimension-agnostic setups (ndim=1/2/3 from command line) +- Load parameters from JSON/YAML/HDF5 +- Generate parameter sweeps from environment variables + +**3. Flexible Abstractions** +- Use provided helper classes (`InputFile`, `Block`) +- Write your own JSON/YAML parsers +- Direct API usage with no abstractions +- Application-specific parameter generators + +## Design Philosophy + +**Minimal Core, Flexible Tooling**: The C++ infrastructure only provides: +1. Embedded Python interpreter (via pybind11) +2. ParameterInput bindings (`add_int`, `add_real`, `add_bool`, `add_string`, `add_*_vector`) +3. Explicit function call pattern: Python files define `parthenon_init_parameters(pin)` + +Users can write their own Python abstractions to suit their needs: +- Helper classes like `InputFile` and `Block` (provided as an example) +- JSON/YAML parsers that populate ParameterInput +- Direct scripting without abstractions +- Application-specific parameter generators + +## Architecture + +### C++ Side (Required Core) + +``` +┌─────────────────────────────────────────────┐ +│ parthenon_manager.cpp │ +│ - Detect .py extension │ +│ - Call LoadParameterInputFromPython() │ +└─────────────────┬───────────────────────────┘ + │ + ↓ +┌─────────────────────────────────────────────┐ +│ parameter_parsers/python_parser.cpp │ +│ - Start embedded Python interpreter │ +│ - Set sys.argv for script │ +│ - Execute .py file (load function defs) │ +│ - Call parthenon_init_parameters(pin) │ +└─────────────────┬───────────────────────────┘ + │ + ↓ +┌─────────────────────────────────────────────┐ +│ pybind/parameter_input_bindings.cpp │ +│ - Expose add_int, add_real, add_bool, ... │ +│ - Expose add_*_vector methods │ +│ - Expose query methods (does_exist, etc.) │ +└─────────────────────────────────────────────┘ +``` + +### Python Side (Optional Tooling) + +``` +┌─────────────────────────────────────────────┐ +│ User's input.py script │ +│ - Define: parthenon_init_parameters(pin) │ +│ - Inside function: │ +│ - pin.add_int("block", "param", value) │ +│ - pin.add_real(...) │ +│ │ +│ Optional: use helper abstractions │ +│ - from parthenon_input import InputFile │ +│ - inp = InputFile() │ +│ - inp.block("mesh", nx1=64, nx2=64) │ +│ - inp.to_parameter_input(pin) │ +└─────────────────────────────────────────────┘ +``` + +## Core Implementation + +### File Extension Detection + +```cpp +// parthenon_manager.cpp +bool is_python_input = (fs::path(arg.input_filename).extension() == ".py"); + +#ifdef PARTHENON_ENABLE_PYTHON_BINDINGS + if (is_python_input) { + pinput = LoadParameterInputFromPython(arg.input_filename, argc, argv); + } +#else + if (is_python_input) { + PARTHENON_FAIL("Python input detected but not enabled at build time"); + } +#endif +``` + +### Python Interpreter Lifecycle + +```cpp +// parameter_parsers/python_parser.cpp +std::unique_ptr LoadParameterInputFromPython( + const char *python_filename, int argc, char *argv[]) { + + auto pinput = std::make_unique(); + + py::scoped_interpreter guard{}; // Start interpreter + + // Import parthenon module (makes bindings available) + py::module_::import("parthenon"); + + // Build sys.argv for the script (includes arguments after -i) + py::list py_argv; + py_argv.append(python_filename); + // ... add remaining arguments ... + py::module_::import("sys").attr("argv") = py_argv; + + // Execute Python file to load function definitions + py::dict globals = py::globals(); + py::eval_file(python_filename, globals); + + // Look for required initialization function + if (!globals.contains("parthenon_init_parameters")) { + PARTHENON_FAIL("Python script must define parthenon_init_parameters(pin)"); + } + + py::object init_func = globals["parthenon_init_parameters"]; + + // Call initialization function with ParameterInput + init_func(py::cast(pinput.get(), py::return_value_policy::reference)); + + return pinput; // Interpreter destroyed, ParameterInput returned to C++ +} +``` + +### Python Bindings (Minimal API) + +```cpp +// pybind/parameter_input_bindings.cpp +PYBIND11_MODULE(parthenon, m) { + py::class_(m, "ParameterInput") + // Add methods (typed, parser interface) + .def("add_int", &parthenon::ParameterInput::AddParsedParameter) + .def("add_real", &parthenon::ParameterInput::AddParsedParameter) + .def("add_bool", &parthenon::ParameterInput::AddParsedParameter) + .def("add_string", &parthenon::ParameterInput::AddParsedParameter) + .def("add_int_vector", &parthenon::ParameterInput::AddParsedParameter>) + .def("add_real_vector", &parthenon::ParameterInput::AddParsedParameter>) + .def("add_bool_vector", &parthenon::ParameterInput::AddParsedParameter>) + .def("add_string_vector", &parthenon::ParameterInput::AddParsedParameter>) + .def("add_unresolved", /* for parameters from nested .pin files */) + + // Query methods (safe during parsing, don't trigger finalization) + .def("does_parameter_exist", &parthenon::ParameterInput::DoesParameterExist) + .def("does_block_exist", &parthenon::ParameterInput::DoesBlockExist) + .def("get_parameter_names", &parthenon::ParameterInput::GetParameterNames) + .def("get_blocks_with_prefix", &parthenon::ParameterInput::GetBlocksWithPrefix); + + // ParameterInput is passed explicitly to parthenon_init_parameters(pin) + // No global injection or get_parameter_input() function needed + + // Note: Get methods (get_int, get_real, etc.) are NOT exposed during parsing. + // They trigger FinalizeParsing(), which would break ModifyFromCmdline(). + // Python scripts should only ADD parameters during the parsing phase. +} +``` + +## Usage Patterns + +### Pattern 1: Direct API (No Helper Classes) + +```python +#!/usr/bin/env python3 + +def parthenon_init_parameters(pin): + """Configure parameters using direct API.""" + # Add parameters directly + pin.add_int("parthenon/mesh", "nx1", 64) + pin.add_int("parthenon/mesh", "nx2", 64) + pin.add_int("parthenon/mesh", "nx3", 1) + pin.add_real("parthenon/mesh", "x1min", 0.0) + pin.add_real("parthenon/mesh", "x1max", 1.0) + + pin.add_real("parthenon/time", "tlim", 1.0) + pin.add_int("parthenon/time", "nlim", 100) +``` + +### Pattern 2: With Helper Classes (Optional) + +```python +#!/usr/bin/env python3 +from parthenon_input import InputFile + +def parthenon_init_parameters(pin): + """Configure using helper class.""" + inp = InputFile() + inp.block("parthenon/mesh", nx1=64, nx2=64, nx3=1) + inp.block("parthenon/time", tlim=1.0, nlim=100) + inp.to_parameter_input(pin) +``` + +### Pattern 3: From JSON (User-Written) + +```python +#!/usr/bin/env python3 +import json + +def parthenon_init_parameters(pin): + """Populate parameters from JSON.""" + with open("config.json") as f: + config = json.load(f) + + for block_name, params in config.items(): + for key, value in params.items(): + if isinstance(value, int): + pin.add_int(block_name, key, value) + elif isinstance(value, float): + pin.add_real(block_name, key, value) + # ... etc +``` + +### Pattern 4: Programmatic Generation + +```python +#!/usr/bin/env python3 +import argparse + +def parthenon_init_parameters(pin): + """Generate parameters programmatically based on command line args.""" + parser = argparse.ArgumentParser() + parser.add_argument("--ndim", type=int, default=2) + parser.add_argument("--nx", type=int, default=64) + args, _ = parser.parse_known_args() + + # Set mesh based on dimensionality + pin.add_int("parthenon/mesh", "nx1", args.nx) + pin.add_int("parthenon/mesh", "nx2", args.nx if args.ndim >= 2 else 1) + pin.add_int("parthenon/mesh", "nx3", args.nx if args.ndim >= 3 else 1) +``` + +## Command Line Argument Handling + +Python scripts receive arguments via `sys.argv`: + +```bash +./myapp -i input.py --nx=128 parthenon/mesh/refinement=static +``` + +```python +# input.py sees: ["input.py", "--nx=128", "parthenon/mesh/refinement=static"] +import argparse + +def parthenon_init_parameters(pin): + """Parse arguments and configure parameters.""" + # Parse Python-style arguments + parser = argparse.ArgumentParser() + parser.add_argument("--nx", type=int, default=64) + args, remaining = parser.parse_known_args() # remaining = ["parthenon/mesh/refinement=static"] + + pin.add_int("parthenon/mesh", "nx1", args.nx) + # Function completes... + +# C++ then processes remaining Parthenon-style arguments via ModifyFromCmdline() +# This sets parthenon/mesh/refinement = "static" (overriding any Python value) +``` + +### Handling --help and Clean Exits + +When a Python script calls `sys.exit(0)` (e.g., argparse with `--help`), the application exits cleanly: + +```bash +./myapp -i input.py --help +``` + +- Exit code 0: Clean exit, application returns immediately without starting simulation +- Exit code non-zero: Fatal error with error message + +This allows Python scripts to provide rich help text via argparse. + +## Provided Tooling (Optional) + +The `parthenon_input` package provides **example** helper classes: + +### InputFile Class + +```python +class InputFile: + """Accumulator for parameter blocks.""" + def block(self, name, **params): + """Add a parameter block.""" + blk = Block(name, **params) + self.blocks.append(blk) + return blk + + def to_parameter_input(self, pi=None): + """Transfer to C++ ParameterInput with type preservation.""" + for block in self.blocks: + for key, value in block.params.items(): + # Dispatch based on Python type + if isinstance(value, bool): + pi.add_bool(block.name, key, value) + elif isinstance(value, int): + pi.add_int(block.name, key, value) + # ... etc +``` + +**Note**: This is just one possible abstraction. Users can write their own. + +## Build System + +```cmake +# src/pybind/CMakeLists.txt +pybind11_add_module(parthenon_py parameter_input_bindings.cpp) +set_target_properties(parthenon_py PROPERTIES + OUTPUT_NAME parthenon + LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib/python) + +# Create symlinks to Python packages for convenient PYTHONPATH +add_custom_command(TARGET parthenon_py POST_BUILD + COMMAND ${CMAKE_COMMAND} -E create_symlink + ${PYTHON_PACKAGES_DIR}/parthenon_input + ${PYTHON_LIB_DIR}/parthenon_input) +``` + +Single PYTHONPATH entry: +```bash +export PYTHONPATH=/path/to/build/lib/python:$PYTHONPATH +``` + +## Integration with Core Refactor + +The Python parser integrates seamlessly with the parser separation refactor: + +1. Python script executes, populates ParameterInput via `add_*` methods +2. Interpreter shuts down, returns populated ParameterInput to C++ +3. C++ applies command line overrides via `ModifyFromCmdline()` +4. C++ calls `FinalizeParsing()` to mark parsing complete +5. Application queries parameters via `Get()` and `GetOrAdd()` + +The Python script runs **before** `FinalizeParsing()`, so it uses the same parser interface as text files. + +## Why Get Methods Are Not Exposed + +The Python bindings intentionally **do not** expose `Get()` methods: + +```cpp +// NOT exposed: +// .def("get_int", &parthenon::ParameterInput::Get) +``` + +**Reason**: `Get()` triggers `FinalizeParsing()`, which would prevent `ModifyFromCmdline()` from working. Python scripts should only **add** parameters, not query them. If a script needs conditional logic based on existing parameters, use `does_parameter_exist()` and maintain state in Python variables. + +## Build Requirements + +- CMake option: `-DPARTHENON_ENABLE_PYTHON_BINDINGS=ON` +- pybind11 (found via `find_package(pybind11)`) +- Python 3 development headers + +Without Python support: +- `.py` input files trigger clear error message +- No runtime dependency on Python +- All code guarded by `#ifdef PARTHENON_ENABLE_PYTHON_BINDINGS` + +## File Organization + +``` +src/ + parameter_parsers/ + python_parser.hpp # LoadParameterInputFromPython() declaration + python_parser.cpp # Embedded interpreter logic + pybind/ + CMakeLists.txt # Python module build + parameter_input_bindings.cpp # pybind11 bindings + README.md # Python API documentation + +scripts/python/packages/ + parthenon_input/ # Optional helper classes (example tooling) + __init__.py + input_generator.py # InputFile, Block classes + +example/fine_advection/ + parthinput.advection.py # Example Python input file +``` + +## Lines of Code + +**C++ Core (~250 lines total)**: +- `python_parser.cpp`: ~90 lines (interpreter lifecycle) +- `parameter_input_bindings.cpp`: ~135 lines (pybind11 bindings) +- `parthenon_manager.cpp`: ~5 lines (call LoadParameterInputFromPython) +- `CMakeLists.txt`: ~30 lines (build configuration) + +**Python Tooling (~420 lines, optional)**: +- `input_generator.py`: ~377 lines (InputFile, Block classes) +- `__init__.py`: ~40 lines (exports, mpi_print helper) + +Most complexity is in **optional** Python tooling, not the C++ core. + +## Extensibility + +The minimal core enables diverse use cases: + +### JSON Input +```python +import json, parthenon +config = json.load(open("config.json")) +# ... populate ParameterInput from config dict ... +``` + +### YAML Input +```python +import yaml + +def parthenon_init_parameters(pin): + """Load configuration from YAML file.""" + config = yaml.safe_load(open("config.yaml")) + # ... populate pin from config dict ... +``` + +### Parameter Sweeps +```python +import os + +def parthenon_init_parameters(pin): + """Configure parameters based on environment variables.""" + run_id = int(os.environ.get("RUN_ID", 0)) + pin.add_real("problem", "amplitude", 0.1 * (run_id + 1)) +``` + +### Application-Specific Abstractions +```python +# User writes their own abstractions +from my_app_utils import ProblemSetup + +def parthenon_init_parameters(pin): + """Use application-specific configuration helper.""" + setup = ProblemSetup(mode="standard", param_a=1.4, param_b=64) + setup.configure_parameter_input(pin) +``` + +## Testing + +- Unit tests: Python bindings tested via pytest (if desired) +- Regression tests: Example `parthinput.advection.py` runs with fine_advection +- Build tests: Both `-DPARTHENON_ENABLE_PYTHON_BINDINGS=ON` and `OFF` configurations + +## Summary + +This implementation provides a **minimal, flexible foundation** for Python input support: + +1. **Core (C++)**: Just enough to embed Python and expose ParameterInput methods +2. **Tooling (Python)**: Optional abstractions that users can replace or extend +3. **No vendor lock-in**: Users can write their own JSON parsers, abstractions, etc. +4. **Clean integration**: Works seamlessly with parser separation refactor +5. **Backward compatible**: Text input still works, Python is purely additive + +The philosophy is: provide the plumbing, let users build their own faucets. diff --git a/example/fine_advection/README.md b/example/fine_advection/README.md index 8e09550c8a94f..6fad59707cc6e 100644 --- a/example/fine_advection/README.md +++ b/example/fine_advection/README.md @@ -1,5 +1,86 @@ -This example implements upwind advection of a cell-centered scalar variable defined + + +This example implements upwind advection of a cell-centered scalar variable defined on the regular grid and for another cell-centered variable on the fine grid (which is twice the resolution and is selected using Metadata::Fine). The newer type-based -`SparsePack`s are used throughout and machinery for doing a generalized Stoke's -theorem based update is included. \ No newline at end of file +`SparsePack`s are used throughout and machinery for doing a generalized Stoke's +theorem based update is included. + +## Running the example + +The example can be run with either a traditional text input file or a Python input file: + +```bash +# Using text input file +./fine_advection-example -i parthinput.advection + +# Using Python input file (requires -DPARTHENON_ENABLE_PYTHON_BINDINGS=ON) +# From the repo root: +PYTHONPATH=build/lib/python \ + build/example/fine_advection/fine_advection-example -i example/fine_advection/parthinput.advection.py + +# Or export PYTHONPATH once: +export PYTHONPATH=/path/to/parthenon/build/lib/python:$PYTHONPATH +./fine_advection-example -i parthinput.advection.py +``` + +### PYTHONPATH requirements + +Python input files require: +- `parthenon` module - Python bindings (built in `build/lib/python/`) +- `parthenon_input` package - Input file helpers (symlinked to `build/lib/python/`) +- `parthenon_tools` package - Analysis utilities (symlinked to `build/lib/python/`) + +The build system automatically creates `build/lib/python/` with all components. +After `make install`, they will be in `CMAKE_INSTALL_PREFIX/lib/python/`. + +### MPI considerations + +Python input files run **independently on every MPI rank**. Each rank: +- Starts its own Python interpreter +- Executes the entire script +- Configures its own ParameterInput object + +To print only from rank 0, use the `mpi_print` helper: + +```python +from parthenon_input import mpi_print +mpi_print(f"Configured {ndim}D problem with resolution {nx}") +``` + +This behaves exactly like `print()` but only outputs from rank 0. For more complex +rank-specific operations, check `parthenon.my_rank` and `parthenon.nranks`. + +Best practices: +- Keep scripts deterministic (same parameters on all ranks) +- Use `mpi_print()` instead of `print()` to avoid output spam +- Avoid file I/O unless coordinated (or use rank-specific filenames) +- Don't use random numbers without setting seed based on rank + +### Python input advantages + +The Python input file (`parthinput.advection.py`) demonstrates several advantages over text files: +- **Command line arguments**: Pass Python-style flags like `--nx=128 --ndim=3` +- **Dimensionality control**: Change `ndim` to easily switch between 1D, 2D, or 3D +- **Calculated parameters**: Automatically derive `derefine_tol` from `refine_tol` +- **Variables**: Define resolution once, use everywhere +- **Type safety**: Use native Python types (lists, not comma-separated strings) +- **Documentation**: Inline comments explaining parameter choices + +### Command line arguments + +Python input files support both Python-style and Parthenon-style arguments: + +```bash +# Python-style arguments (parsed by the script) +./fine_advection-example -i parthinput.advection.py --nx=128 --ndim=3 + +# Parthenon-style overrides (processed by C++ after Python runs) +./fine_advection-example -i parthinput.advection.py parthenon/time/tlim=0.5 + +# Both can be combined +./fine_advection-example -i parthinput.advection.py --nx=128 parthenon/time/tlim=0.5 +``` + +Python scripts use `argparse.parse_known_args()` to parse their own flags while ignoring +Parthenon-style overrides, which are applied by C++ after the script completes. \ No newline at end of file diff --git a/example/fine_advection/parthinput.advection.py b/example/fine_advection/parthinput.advection.py new file mode 100644 index 0000000000000..bdcdc4ef77593 --- /dev/null +++ b/example/fine_advection/parthinput.advection.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +# ======================================================================================== +# (C) (or copyright) 2020-2026. Triad National Security, LLC. All rights reserved. +# +# This program was produced under U.S. Government contract 89233218CNA000001 for Los +# Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC +# for the U.S. Department of Energy/National Nuclear Security Administration. All rights +# in the program are reserved by Triad National Security, LLC, and the U.S. Department +# of Energy/National Nuclear Security Administration. The Government is granted for +# itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide +# license in this material to reproduce, prepare derivative works, distribute copies to +# the public, perform publicly and display publicly, and to permit others to do so. +# ======================================================================================== +# This file was made in part with generative AI. + +# Example Python input file for fine_advection example +# Demonstrates advantages over text input files: +# - Single file works for 1D, 2D, or 3D (just change ndim) +# - Command line arguments for easy parameter studies +# - Use variables and calculations +# - Compute derived quantities automatically +# - Cleaner than duplicating parameters across dimensions + +import argparse +from parthenon_input import InputFile, mpi_print + + +def parthenon_init_parameters(pin): + """Configure parameters for fine advection example. + + Args: + pin: ParameterInput object to configure + """ + parser = argparse.ArgumentParser(description="Fine advection example") + parser.add_argument( + "--ndim", type=int, default=2, help="Number of dimensions (1, 2, or 3)" + ) + parser.add_argument( + "--nx", type=int, default=64, help="Base mesh resolution (in active dimensions)" + ) + parser.add_argument( + "--meshblock-size", + type=int, + default=16, + help="Meshblock size (in active dimensions)", + ) + parser.add_argument("--num-levels", type=int, default=3, help="Number of AMR levels") + parser.add_argument("--cfl", type=float, default=0.45, help="CFL number") + args, unknown = parser.parse_known_args() + + # ====================================================================================== + # PROBLEM CONFIGURATION + # ====================================================================================== + ndim = args.ndim + nx_base = args.nx + meshblock_size = args.meshblock_size + num_amr_levels = args.num_levels + cfl = args.cfl + + # Fixed parameters + domain_min = -0.5 # Domain bounds + domain_max = 0.5 + velocity = 1.0 # Advection velocity (in active dimensions) + refine_tol = 0.3 # AMR refinement tolerance + output_dt = 0.05 # Output cadence + + # ====================================================================================== + # BUILD CONFIGURATION (automatic based on ndim) + # ====================================================================================== + inp = InputFile() + + inp.block("parthenon/job", problem_id="advection") + + # Mesh configuration - automatically set dimensions based on ndim + inp.block( + "parthenon/mesh", + refinement="adaptive", + numlevel=num_amr_levels, + # Dimension 1 (always active) + nx1=nx_base, + x1min=domain_min, + x1max=domain_max, + ix1_bc="periodic", + ox1_bc="periodic", + # Dimension 2 (active if ndim >= 2) + nx2=nx_base if ndim >= 2 else 1, + x2min=domain_min, + x2max=domain_max, + ix2_bc="periodic", + ox2_bc="periodic", + # Dimension 3 (active if ndim >= 3) + nx3=nx_base if ndim >= 3 else 1, + x3min=domain_min, + x3max=domain_max, + ix3_bc="periodic", + ox3_bc="periodic", + ) + + # Meshblock configuration - automatically sized based on ndim + inp.block( + "parthenon/meshblock", + nx1=meshblock_size, + nx2=meshblock_size if ndim >= 2 else 1, + nx3=meshblock_size if ndim >= 3 else 1, + ) + + inp.block("parthenon/time", nlim=-1, tlim=1.0, integrator="rk2", ncycle_out_mesh=-10000) + + # Advection parameters - velocities set based on ndim + inp.block( + "Advection", + cfl=cfl, + vx=velocity, + vy=velocity if ndim >= 2 else 0.0, + vz=velocity if ndim >= 3 else 0.0, + profile="hard_sphere", + # Automatically compute derefine tolerance + refine_tol=refine_tol, + derefine_tol=refine_tol / 10.0, + # Feature flags + do_regular_advection=True, + do_fine_advection=True, + do_CT_advection=True, + ) + + # Restart output + inp.block("parthenon/output1", file_type="rst", dt=output_dt) + + # HDF5 output + inp.block( + "parthenon/output0", + file_type="hdf5", + dt=output_dt, + variables=["advection.scalar", "advection.scalar_fine_restricted"], + ) + + # Transfer all configuration to C++ ParameterInput + inp.to_parameter_input(pin) + + # Print summary only from rank 0 using mpi_print helper + mpi_print(f"Configured {ndim}D advection problem:") + mpi_print(f" Resolution: {nx_base}^{ndim}") + mpi_print(f" Meshblock size: {meshblock_size}") + mpi_print(f" AMR levels: {num_amr_levels}") + mpi_print(f" CFL: {cfl}") diff --git a/scripts/python/packages/parthenon_input/__init__.py b/scripts/python/packages/parthenon_input/__init__.py new file mode 100644 index 0000000000000..a723cddd1dd7a --- /dev/null +++ b/scripts/python/packages/parthenon_input/__init__.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +# ========================================================================================= +# (C) (or copyright) 2020-2026. Triad National Security, LLC. All rights reserved. +# +# This program was produced under U.S. Government contract 89233218CNA000001 for Los +# Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC +# for the U.S. Department of Energy/National Nuclear Security Administration. All rights +# in the program are reserved by Triad National Security, LLC, and the U.S. Department +# of Energy/National Nuclear Security Administration. The Government is granted for +# itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide +# license in this material to reproduce, prepare derivative works, distribute copies to +# the public, perform publicly and display publicly, and to permit others to do so. +# ========================================================================================= +# This file was made in part with generative AI. + +from .input_generator import InputFile, Block + + +def mpi_print(*args, **kwargs): + """Print only from MPI rank 0. + + Behaves exactly like built-in print(), but only outputs from rank 0. + Useful in Python input files to avoid duplicated output. + + Example: + from parthenon_input import mpi_print + mpi_print(f"Configured {ndim}D problem with resolution {nx}") + """ + try: + import parthenon + + if parthenon.my_rank() == 0: + print(*args, **kwargs) + except (ImportError, AttributeError, TypeError): + # If parthenon module not available, my_rank not set, or not callable, + # just print (e.g., when running outside of embedded context) + print(*args, **kwargs) + + +__all__ = ["InputFile", "Block", "mpi_print"] diff --git a/scripts/python/packages/parthenon_input/input_generator.py b/scripts/python/packages/parthenon_input/input_generator.py new file mode 100644 index 0000000000000..b2fed32a4815a --- /dev/null +++ b/scripts/python/packages/parthenon_input/input_generator.py @@ -0,0 +1,377 @@ +#!/usr/bin/env python3 +# ========================================================================================= +# (C) (or copyright) 2020-2026. Triad National Security, LLC. All rights reserved. +# +# This program was produced under U.S. Government contract 89233218CNA000001 for Los +# Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC +# for the U.S. Department of Energy/National Nuclear Security Administration. All rights +# in the program are reserved by Triad National Security, LLC, and the U.S. Department +# of Energy/National Nuclear Security Administration. The Government is granted for +# itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide +# license in this material to reproduce, prepare derivative works, distribute copies to +# the public, perform publicly and display publicly, and to permit others to do so. +# ========================================================================================= +# This file was made in part with generative AI. + +""" +Python-based input file generator for Parthenon. + +Provides a two-stage approach: +1. Build mutable parameter structure in Python +2. Transfer to typed C++ ParameterInput when ready + +Example usage: + from parthenon_input import InputFile + + inp = InputFile() + mesh = inp.block("parthenon/mesh", nx1=64, nx2=64, nx3=64) + mesh.params["x1min"] = 0.0 # Can modify after creation + + inp.block("parthenon/time", tlim=1.0, nlim=100) + + # Transfer to C++ ParameterInput with type preservation + pi = inp.to_parameter_input() +""" + +from typing import Any, Dict, List, Optional + + +class Block: + """Represents a parameter block in a Parthenon input file.""" + + def __init__(self, name: str, **params: Any): + """ + Create a parameter block. + + Args: + name: Block name (e.g., "parthenon/mesh") + **params: Parameter key-value pairs + """ + self.name = name + self.params = {} + self._typed = {} # Track which params are typed (True) vs from file (False) + + for key, value in params.items(): + self.params[key] = value + self._typed[key] = True # Parameters passed at construction are typed + + def set(self, **params: Any) -> None: + """ + Set or update parameters with strong typing. + + Args: + **params: Parameter key-value pairs to set + + Example: + block.set(nx1=128, nx2=128, x1min=0.0) + """ + for key, value in params.items(): + self.params[key] = value + self._typed[key] = True # Explicitly set parameters are typed + + def _set_from_file(self, key: str, value: str) -> None: + """Internal: Set parameter from file (unresolved string).""" + self.params[key] = value + self._typed[key] = False # From file, will need lazy conversion + + def is_typed(self, key: str) -> bool: + """Check if a parameter is strongly typed (vs loaded from file).""" + return self._typed.get(key, True) + + def _format_value(self, value: Any) -> str: + """Convert a Python value to input file format.""" + if isinstance(value, bool): + return "true" if value else "false" + elif isinstance(value, (list, tuple)): + # Handle vectors + return ", ".join(str(v) for v in value) + elif isinstance(value, str): + return value + else: + return str(value) + + def to_string(self) -> str: + """Convert block to input file format.""" + lines = [f"<{self.name}>"] + for key, value in self.params.items(): + formatted = self._format_value(value) + lines.append(f"{key} = {formatted}") + return "\n".join(lines) + + +class InputFile: + """ + Accumulator for parameter blocks that can transfer to C++ ParameterInput. + + This provides a two-stage approach: + 1. Build mutable parameter structure in Python + 2. Transfer to typed C++ ParameterInput when ready + + Example: + inp = InputFile() + mesh = inp.block("parthenon/mesh", nx1=64, nx2=64) + mesh.params["x1min"] = 0.0 # Can modify after creation + + # Transfer to C++ with type preservation + pi = inp.to_parameter_input() + """ + + def __init__(self, header: Optional[str] = None): + """ + Create an input file builder. + + Args: + header: Optional comment header to include at top of file + """ + self.blocks: List[Block] = [] + self.header = header + + def block(self, name: str, **params: Any) -> Block: + """ + Add a parameter block and return it for further modification. + + Args: + name: Block name (e.g., "parthenon/mesh") + **params: Parameter key-value pairs + + Returns: + Block object (already added to this InputFile) + + Example: + inp = InputFile() + mesh = inp.block("parthenon/mesh", nx1=64) + mesh.set(nx2=128) # Can modify after creation + """ + blk = Block(name, **params) + self.blocks.append(blk) + return blk + + def get_block(self, name: str) -> Optional[Block]: + """ + Get a block by name. + + Args: + name: Block name (e.g., "parthenon/mesh") + + Returns: + Block object if found, None otherwise + + Example: + mesh = inp.get_block("parthenon/mesh") + if mesh: + mesh.set(nx1=128) + """ + for block in self.blocks: + if block.name == name: + return block + return None + + def to_parameter_input(self, pi=None): + """ + Transfer to C++ ParameterInput with full type preservation. + + This dispatches each parameter to the appropriate AddParsedParameter() method + based on its Python type: + - int -> add_int() + - float -> add_real() + - bool -> add_bool() + - str -> add_string() + - list[int] -> add_int_vector() + - etc. + + Args: + pi: Optional ParameterInput object to populate. If None, creates a new one. + + Returns: + Pybind11-wrapped ParameterInput object (either provided or newly created) + + Example: + inp = InputFile() + inp.block("parthenon/mesh", nx1=64, x1min=0.0) + pi = inp.to_parameter_input() + + Example with existing ParameterInput: + # pi provided by C++ code + inp = InputFile() + inp.block("parthenon/mesh", nx1=64) + inp.to_parameter_input(pi) # populate existing pi + """ + try: + import parthenon + except ImportError: + raise ImportError( + "parthenon module not found. " + "Make sure pybind11 bindings are built and installed." + ) + + if pi is None: + pi = parthenon.ParameterInput() + + for block in self.blocks: + for key, value in block.params.items(): + is_typed = block.is_typed(key) + self._add_typed_parameter(pi, block.name, key, value, is_typed) + + # NOTE: Don't call finalize_parsing() here - application may still want to + # call ModifyFromCmdline() or other parsing. Application should call + # finalize_parsing() or let first Get/GetOrAdd call it automatically. + + return pi + + def _add_typed_parameter( + self, pi, block_name: str, param_name: str, value: Any, is_typed: bool = True + ): + """ + Dispatch to appropriate AddParsedParameter method based on parameter type and origin. + + Args: + pi: C++ ParameterInput object + block_name: Block name + param_name: Parameter name + value: Parameter value + is_typed: If False, value is from file and should use UnresolvedString + """ + # If parameter came from file, use unresolved string for lazy conversion + if not is_typed: + pi.add_unresolved(block_name, param_name, str(value)) + return + + # Otherwise dispatch based on Python type + if isinstance(value, bool): + # Must check bool before int (bool is subclass of int in Python) + pi.add_bool(block_name, param_name, value) + elif isinstance(value, int): + pi.add_int(block_name, param_name, value) + elif isinstance(value, float): + pi.add_real(block_name, param_name, value) + elif isinstance(value, str): + pi.add_string(block_name, param_name, value) + elif isinstance(value, (list, tuple)): + # Dispatch vector based on element type + if len(value) == 0: + raise ValueError( + f"Cannot infer type of empty list for {block_name}/{param_name}" + ) + first = value[0] + if isinstance(first, bool): + pi.add_bool_vector(block_name, param_name, list(value)) + elif isinstance(first, int): + pi.add_int_vector(block_name, param_name, list(value)) + elif isinstance(first, float): + pi.add_real_vector(block_name, param_name, list(value)) + elif isinstance(first, str): + pi.add_string_vector(block_name, param_name, list(value)) + else: + raise TypeError(f"Unsupported vector element type: {type(first)}") + else: + raise TypeError(f"Unsupported parameter type: {type(value)}") + + def __str__(self) -> str: + """ + Generate text representation (for debugging or fallback). + + Note: For production use, prefer to_parameter_input() which preserves types. + """ + lines = [] + + if self.header: + for line in self.header.split("\n"): + lines.append(f"# {line}") + lines.append("") + + for i, blk in enumerate(self.blocks): + lines.append(blk.to_string()) + # Add blank line between blocks (but not after last one) + if i < len(self.blocks) - 1: + lines.append("") + + return "\n".join(lines) + "\n" + + def write(self, filename: str) -> None: + """ + Write text representation to disk (for debugging or fallback). + + Note: For production use, prefer to_parameter_input() which preserves types. + + Args: + filename: Output filename + """ + with open(filename, "w") as f: + f.write(str(self)) + + @staticmethod + def from_dict( + config: Dict[str, Dict[str, Any]], header: Optional[str] = None + ) -> "InputFile": + """ + Create an InputFile from a nested dictionary. + + Useful for loading from JSON/YAML. + + Args: + config: Dictionary of block_name -> {param: value} + header: Optional comment header + + Returns: + InputFile object + + Example: + config = { + "parthenon/mesh": {"nx1": 64, "nx2": 64}, + "parthenon/time": {"tlim": 1.0} + } + inp = InputFile.from_dict(config) + pi = inp.to_parameter_input() + """ + inp = InputFile(header=header) + for name, params in config.items(): + inp.block(name, **params) + return inp + + @staticmethod + def from_file(filename: str) -> "InputFile": + """ + Load an existing Parthenon input file. + + Parameters loaded from file are stored as unresolved strings + for lazy type conversion (matching C++ behavior). Parameters + subsequently modified in Python become strongly typed. + + Args: + filename: Path to .pin file + + Returns: + InputFile object with parameters loaded from file + + Example: + inp = InputFile.from_file("base.pin") + mesh = inp.get_block("parthenon/mesh") + mesh.set(nx1=128) # This override becomes strongly typed + pi = inp.to_parameter_input() + """ + inp = InputFile() + current_block = None + + with open(filename, "r") as f: + for line in f: + # Strip whitespace and comments + line = line.split("#")[0].strip() + if not line: + continue + + # Check for block header + if line.startswith("<") and line.endswith(">"): + block_name = line[1:-1] + current_block = Block(block_name) + inp.blocks.append(current_block) + continue + + # Parse parameter line + if "=" in line and current_block is not None: + key, value = line.split("=", 1) + key = key.strip() + value = value.strip() + # Store as unresolved string (lazy conversion) + current_block._set_from_file(key, value) + + return inp diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 9962eb48b6244..b9bc15ec21961 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -330,6 +330,8 @@ add_library(parthenon kokkos_abstraction.hpp parameter_input.cpp parameter_input.hpp + parameter_parsers/python_parser.cpp + parameter_parsers/python_parser.hpp parthenon_array_generic.hpp parthenon_arrays.cpp parthenon_arrays.hpp @@ -402,6 +404,15 @@ endif() lint_target(parthenon) +# Python bindings (optional) +add_subdirectory(pybind) + +# Link parthenon against pybind11::embed if Python bindings are enabled +# (needed for embedding Python interpreter in parthenon_manager.cpp) +if(PARTHENON_ENABLE_PYTHON_BINDINGS) + target_link_libraries(parthenon PUBLIC pybind11::embed) +endif() + target_include_directories(parthenon PUBLIC $ $ diff --git a/src/config.hpp.in b/src/config.hpp.in index 3a8834a739c64..59abca280a56a 100644 --- a/src/config.hpp.in +++ b/src/config.hpp.in @@ -58,6 +58,9 @@ // define PARTHENON_USE_SERIAL_POOL or not at all #cmakedefine PARTHENON_USE_SERIAL_POOL +// define PARTHENON_ENABLE_PYTHON_BINDINGS or not at all +#cmakedefine PARTHENON_ENABLE_PYTHON_BINDINGS + // Default loop patterns for MeshBlock par_for() wrappers, // see kokkos_abstraction.hpp for available tags. // Kokkos tight loop layout diff --git a/src/parameter_parsers/python_parser.cpp b/src/parameter_parsers/python_parser.cpp new file mode 100644 index 0000000000000..674aa38127040 --- /dev/null +++ b/src/parameter_parsers/python_parser.cpp @@ -0,0 +1,148 @@ +//======================================================================================== +// (C) (or copyright) 2020-2026. Triad National Security, LLC. All rights reserved. +// +// This program was produced under U.S. Government contract 89233218CNA000001 for Los +// Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC +// for the U.S. Department of Energy/National Nuclear Security Administration. All rights +// in the program are reserved by Triad National Security, LLC, and the U.S. Department +// of Energy/National Nuclear Security Administration. The Government is granted for +// itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide +// license in this material to reproduce, prepare derivative works, distribute copies to +// the public, perform publicly and display publicly, and to permit others to do so. +//======================================================================================== +// This file was made in part with generative AI. + +#include "python_parser.hpp" + +#ifdef PARTHENON_ENABLE_PYTHON_BINDINGS + +#include +#include + +#include +#include +#include + +#include "utils/error_checking.hpp" + +namespace py = pybind11; + +namespace parthenon { + +std::unique_ptr LoadParameterInputFromPython(const char *python_filename, + int argc, char *argv[]) { + // Create ParameterInput in C++ - we own this + auto pinput = std::make_unique(); + + // Start Python interpreter + py::scoped_interpreter guard{}; + + try { + // Import the parthenon module to make ParameterInput bindings available + // The parthenon.so module must be in PYTHONPATH + py::module_::import("parthenon"); + + // Build sys.argv for the Python script + // Include the script name and all command line arguments after "-i script.py" + // This allows Python scripts to use argparse to parse their own arguments + py::list py_argv; + py_argv.append(python_filename); + + // Find where the input file appears in argv and include everything after it + bool found_input_file = false; + for (int i = 1; i < argc; i++) { + if (found_input_file) { + py_argv.append(argv[i]); + } else if (std::string(argv[i]) == "-i" && i + 1 < argc) { + // Skip -i and the filename, start collecting args after + i++; // Skip the filename + found_input_file = true; + } + } + + // Set sys.argv for the Python script + py::module_::import("sys").attr("argv") = py_argv; + + // Execute the Python script to load function definitions + // The script can use argparse.parse_known_args() to parse Python-style flags (e.g., --nx=32) + // After this returns, C++ can apply Parthenon-style overrides (block/param=value) + // via ModifyFromCmdline(), which ignores Python-style flags + py::dict globals = py::globals(); + py::eval_file(python_filename, globals); + + // Look for required initialization function + if (!globals.contains("parthenon_init_parameters")) { + std::stringstream msg; + msg << "### FATAL ERROR loading Python input file: " << python_filename << std::endl + << "Python script must define function: parthenon_init_parameters(pin)" << std::endl + << std::endl + << "Example:" << std::endl + << " def parthenon_init_parameters(pin):" << std::endl + << " pin.add_int(\"block\", \"param\", value)" << std::endl; + PARTHENON_FAIL(msg); + } + + py::object init_func = globals["parthenon_init_parameters"]; + + // Check if it's callable + if (!py::isinstance(init_func)) { + std::stringstream msg; + msg << "### FATAL ERROR loading Python input file: " << python_filename << std::endl + << "'parthenon_init_parameters' exists but is not a function" << std::endl; + PARTHENON_FAIL(msg); + } + + // Call the initialization function with the ParameterInput object + try { + init_func(py::cast(pinput.get(), py::return_value_policy::reference)); + } catch (py::error_already_set &e) { + // Let SystemExit propagate to outer handler (e.g., for --help) + if (e.matches(PyExc_SystemExit)) { + throw; + } + // Re-throw other exceptions with better context + std::stringstream msg; + msg << "### FATAL ERROR in parthenon_init_parameters(): " << python_filename + << std::endl << e.what() << std::endl; + PARTHENON_FAIL(msg); + } + } catch (py::error_already_set &e) { + // Handle SystemExit specially (e.g., from argparse --help) + if (e.matches(PyExc_SystemExit)) { + // Extract exit code from SystemExit exception + py::object exit_code_obj = e.value().attr("code"); + int exit_code = 1; // Default to error if we can't extract code + if (!exit_code_obj.is_none()) { + try { + exit_code = exit_code_obj.cast(); + } catch (...) { + // code might not be an int (could be a string or None), treat as error + exit_code = 1; + } + } + + // If exit code is 0, this is a clean exit (e.g., --help) + // Return nullptr to signal caller to exit cleanly + if (exit_code == 0) { + return nullptr; + } + + // Non-zero exit code is an error + std::stringstream msg; + msg << "### FATAL ERROR: Python script exited with code " << exit_code << std::endl; + PARTHENON_FAIL(msg); + } + + // Other Python exceptions are fatal errors + std::stringstream msg; + msg << "### FATAL ERROR loading Python input file: " << python_filename << std::endl + << e.what() << std::endl; + PARTHENON_FAIL(msg); + } + + return pinput; +} + +} // namespace parthenon + +#endif // PARTHENON_ENABLE_PYTHON_BINDINGS diff --git a/src/parameter_parsers/python_parser.hpp b/src/parameter_parsers/python_parser.hpp new file mode 100644 index 0000000000000..b9e4738f0a85a --- /dev/null +++ b/src/parameter_parsers/python_parser.hpp @@ -0,0 +1,32 @@ +//======================================================================================== +// (C) (or copyright) 2020-2026. Triad National Security, LLC. All rights reserved. +// +// This program was produced under U.S. Government contract 89233218CNA000001 for Los +// Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC +// for the U.S. Department of Energy/National Nuclear Security Administration. All rights +// in the program are reserved by Triad National Security, LLC, and the U.S. Department +// of Energy/National Nuclear Security Administration. The Government is granted for +// itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide +// license in this material to reproduce, prepare derivative works, distribute copies to +// the public, perform publicly and display publicly, and to permit others to do so. +//======================================================================================== +// This file was made in part with generative AI. + +#ifndef PARAMETER_PARSERS_PYTHON_PARSER_HPP_ +#define PARAMETER_PARSERS_PYTHON_PARSER_HPP_ + +#include + +#include "parameter_input.hpp" + +namespace parthenon { + +// Load ParameterInput from a Python script +// The script is executed in an embedded Python interpreter and can use +// parthenon.get_parameter_input() to populate parameters programmatically. +std::unique_ptr LoadParameterInputFromPython(const char *python_filename, + int argc, char *argv[]); + +} // namespace parthenon + +#endif // PARAMETER_PARSERS_PYTHON_PARSER_HPP_ diff --git a/src/parthenon_manager.cpp b/src/parthenon_manager.cpp index c91fb5123fecf..f615477582834 100644 --- a/src/parthenon_manager.cpp +++ b/src/parthenon_manager.cpp @@ -42,6 +42,7 @@ #include "outputs/outputs_package.hpp" #include "outputs/restart.hpp" #include "outputs/restart_hdf5.hpp" +#include "parameter_parsers/python_parser.hpp" #include "utils/error_checking.hpp" #include "utils/utils.hpp" @@ -121,16 +122,38 @@ ParthenonStatus ParthenonManager::ParthenonInitEnv(int argc, char *argv[]) { } // If an input file was provided if (arg.input_filename != nullptr) { - // Modify info read from restart file - if (arg.is_restart) { - IOWrapper infile; - infile.Open(arg.input_filename, IOWrapper::FileMode::read); - pinput->LoadFromFile(infile); - infile.Close(); - - // Populate new object for fresh simulation + // Check if it's a Python input file + bool is_python_input = (fs::path(arg.input_filename).extension() == ".py"); + +#ifdef PARTHENON_ENABLE_PYTHON_BINDINGS + if (is_python_input) { + if (arg.is_restart) { + PARTHENON_FAIL("Python input files cannot be used with restart"); + } + pinput = LoadParameterInputFromPython(arg.input_filename, argc, argv); + // nullptr signals clean exit requested (e.g., --help) + if (!pinput) { + return ParthenonStatus::complete; + } + } else { +#else + if (is_python_input) { + PARTHENON_FAIL("Python input file detected but Parthenon was not built with " + "-DPARTHENON_ENABLE_PYTHON_BINDINGS=ON"); } else { - pinput = std::make_unique(arg.input_filename); +#endif + // Standard .pin file handling + // Modify info read from restart file + if (arg.is_restart) { + IOWrapper infile; + infile.Open(arg.input_filename, IOWrapper::FileMode::read); + pinput->LoadFromFile(infile); + infile.Close(); + + // Populate new object for fresh simulation + } else { + pinput = std::make_unique(arg.input_filename); + } } } diff --git a/src/pybind/CMakeLists.txt b/src/pybind/CMakeLists.txt new file mode 100644 index 0000000000000..006800df36200 --- /dev/null +++ b/src/pybind/CMakeLists.txt @@ -0,0 +1,83 @@ +#======================================================================================== +# (C) (or copyright) 2020-2026. Triad National Security, LLC. All rights reserved. +# +# This program was produced under U.S. Government contract 89233218CNA000001 for Los +# Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC +# for the U.S. Department of Energy/National Nuclear Security Administration. All rights +# in the program are reserved by Triad National Security, LLC, and the U.S. Department +# of Energy/National Nuclear Security Administration. The Government is granted for +# itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide +# license in this material to reproduce, prepare derivative works, distribute copies to +# the public, perform publicly and display publicly, and to permit others to do so. +#======================================================================================== +# This file was made in part with generative AI. + +# Python bindings for Parthenon (optional) +option(PARTHENON_ENABLE_PYTHON_BINDINGS "Build Python bindings for parameter input" OFF) + +if(PARTHENON_ENABLE_PYTHON_BINDINGS) + # Suppress policy warning about deprecated FindPythonInterp/FindPythonLibs + if(POLICY CMP0148) + cmake_policy(SET CMP0148 NEW) + endif() + + # First try to find system-installed pybind11 + find_package(pybind11 CONFIG QUIET) + + if(NOT pybind11_FOUND) + message(STATUS "pybind11 not found in system, fetching from GitHub") + include(FetchContent) + + FetchContent_Declare( + pybind11 + GIT_REPOSITORY https://github.com/pybind/pybind11 + GIT_TAG v2.11.1 + GIT_SHALLOW TRUE + ) + + FetchContent_MakeAvailable(pybind11) + else() + message(STATUS "Using system pybind11: ${pybind11_VERSION}") + endif() + + # Build the Python module + # Note: target name is parthenon_py to avoid collision with main library, + # but OUTPUT_NAME is parthenon so Python imports as "import parthenon" + pybind11_add_module(parthenon_py parameter_input_bindings.cpp) + set_target_properties(parthenon_py PROPERTIES + OUTPUT_NAME parthenon + LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/lib/python + ) + + # Link against parthenon library (use :: to avoid ambiguity) + target_link_libraries(parthenon_py PRIVATE parthenon) + + # Create symlinks to Python packages in build/lib/python for convenient PYTHONPATH + # This allows a single PYTHONPATH entry: export PYTHONPATH=build/lib/python:$PYTHONPATH + set(PYTHON_LIB_DIR ${CMAKE_BINARY_DIR}/lib/python) + set(PYTHON_PACKAGES_DIR ${CMAKE_SOURCE_DIR}/scripts/python/packages) + + # Create lib/python directory if it doesn't exist + file(MAKE_DIRECTORY ${PYTHON_LIB_DIR}) + + # Symlink parthenon_input and parthenon_tools packages + add_custom_command(TARGET parthenon_py POST_BUILD + COMMAND ${CMAKE_COMMAND} -E create_symlink + ${PYTHON_PACKAGES_DIR}/parthenon_input + ${PYTHON_LIB_DIR}/parthenon_input + COMMAND ${CMAKE_COMMAND} -E create_symlink + ${PYTHON_PACKAGES_DIR}/parthenon_tools + ${PYTHON_LIB_DIR}/parthenon_tools + COMMENT "Creating symlinks to Python packages in ${PYTHON_LIB_DIR}" + ) + + # Install the Python module to a standard location + install(TARGETS parthenon_py + LIBRARY DESTINATION ${CMAKE_INSTALL_PREFIX}/lib/python) + + message(STATUS "Python bindings enabled and will be built") + message(STATUS "Python module and packages will be in: ${PYTHON_LIB_DIR}") + message(STATUS "Add to PYTHONPATH: export PYTHONPATH=${CMAKE_BINARY_DIR}/lib/python:\$PYTHONPATH") +else() + message(STATUS "Python bindings disabled (use -DPARTHENON_ENABLE_PYTHON_BINDINGS=ON to enable)") +endif() diff --git a/src/pybind/README.md b/src/pybind/README.md new file mode 100644 index 0000000000000..1e595153730d1 --- /dev/null +++ b/src/pybind/README.md @@ -0,0 +1,97 @@ +# Python Bindings for Parthenon Parameter Input + +This directory contains pybind11 bindings for Parthenon's `ParameterInput` class, enabling typed parameter input from Python without string parsing. + +## Overview + +The Python input system provides a two-stage approach: + +1. **Stage 1: Build mutable parameter structure in Python** + - Use `InputFile` class to accumulate parameters + - Full mutability - modify parameters before transfer + - Cleaner syntax than manual text editing + +2. **Stage 2: Transfer to C++ with type preservation** + - Call `to_parameter_input()` to create typed C++ `ParameterInput` + - Automatic type dispatch based on Python types + - No string parsing - direct typed transfer + +## Building + +To enable Python bindings: + +```bash +cmake -DPARTHENON_ENABLE_PYTHON_BINDINGS=ON .. +make +``` + +The build system will: +1. Try to find system-installed pybind11 +2. If not found, automatically fetch from GitHub +3. Build the `parthenon_py` Python module + +## Usage + +```python +from parthenon_input import InputFile + +# Build parameter structure +inp = InputFile() +mesh = inp.block("parthenon/mesh", nx1=64, nx2=64, nx3=64) +mesh.params["x1min"] = 0.0 # Can modify after creation + +inp.block("parthenon/time", tlim=1.0, nlim=100) +inp.block("problem", velocity=[1.0, 0.5, 0.0], periodic=True) + +# Transfer to C++ with type preservation +pi = inp.to_parameter_input() +``` + +## Python API + +**Adding parameters** (use these in Python input scripts): +- `add_int(block, name, value)` +- `add_real(block, name, value)` +- `add_bool(block, name, value)` +- `add_string(block, name, value)` +- `add_int_vector(block, name, list)` +- `add_real_vector(block, name, list)` +- `add_bool_vector(block, name, list)` +- `add_string_vector(block, name, list)` +- `add_unresolved(block, name, string)` - for lazy conversion + +**Querying structure** (safe, const methods): +- `does_parameter_exist(block, name)` - Check if parameter exists +- `does_block_exist(block)` - Check if block exists +- `get_parameter_names(block)` - List parameters in a block +- `get_blocks_with_prefix(prefix)` - Find blocks matching prefix + +**Note**: Parameter value retrieval (`Get` methods) is intentionally not exposed to prevent premature finalization. Python scripts should only **add** parameters, not query their values. + +## Installation + +After building, add the Python module and packages to your PYTHONPATH: + +```bash +export PYTHONPATH=/path/to/parthenon/build/lib/python:$PYTHONPATH +``` + +The build system automatically creates `build/lib/python/` with: +- The compiled `parthenon` module +- Symlinks to `parthenon_input` and `parthenon_tools` packages + +Or install system-wide: + +```bash +make install +``` + +## Example + +See `example/fine_advection/parthinput.advection.py` for a complete working example. + +## Dependencies + +- pybind11 (automatically fetched if not found) +- Python 3.6+ +- parthenon_input Python package diff --git a/src/pybind/parameter_input_bindings.cpp b/src/pybind/parameter_input_bindings.cpp new file mode 100644 index 0000000000000..bf03c783f0cfa --- /dev/null +++ b/src/pybind/parameter_input_bindings.cpp @@ -0,0 +1,148 @@ +//======================================================================================== +// (C) (or copyright) 2020-2026. Triad National Security, LLC. All rights reserved. +// +// This program was produced under U.S. Government contract 89233218CNA000001 for Los +// Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC +// for the U.S. Department of Energy/National Nuclear Security Administration. All rights +// in the program are reserved by Triad National Security, LLC, and the U.S. Department +// of Energy/National Nuclear Security Administration. The Government is granted for +// itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide +// license in this material to reproduce, prepare derivative works, distribute copies to +// the public, perform publicly and display publicly, and to permit others to do so. +//======================================================================================== +// This file was made in part with generative AI. + +#include +#include + +#include "globals.hpp" +#include "parameter_input.hpp" + +namespace py = pybind11; + +// Functions that query MPI directly rather than using Globals +// Note: We can't use Globals::my_rank here because the Python shared library (.so) +// gets its own copy of the static variables when linking against libparthenon.a, +// separate from the executable's copy. When MPI_Init sets Globals::my_rank in the +// executable, the Python module's copy remains at 0. Calling MPI functions directly +// avoids this issue. +int GetMyRank() { +#ifdef MPI_PARALLEL + int rank; + MPI_Comm_rank(MPI_COMM_WORLD, &rank); + return rank; +#else + return 0; +#endif +} + +int GetNRanks() { +#ifdef MPI_PARALLEL + int nranks; + MPI_Comm_size(MPI_COMM_WORLD, &nranks); + return nranks; +#else + return 1; +#endif +} + +PYBIND11_MODULE(parthenon, m) { + m.doc() = "Parthenon Python bindings for parameter input"; + + // Expose MPI rank info as functions that query MPI directly + m.def("my_rank", &GetMyRank, "Get the current MPI rank"); + m.def("nranks", &GetNRanks, "Get the total number of MPI ranks"); + + py::class_(m, "ParameterInput") + .def(py::init<>()) + + // Parser interface - add parameters without creating QueryRecords + .def( + "add_unresolved", + [](parthenon::ParameterInput &self, const std::string &block, + const std::string &name, const std::string &value) { + self.AddParsedParameter(block, name, + parthenon::ParameterInput::UnresolvedString(value)); + }, + "Add a parameter as unresolved string (from file)") + + .def( + "add_int", + [](parthenon::ParameterInput &self, const std::string &block, + const std::string &name, + int value) { self.AddParsedParameter(block, name, value); }, + "Add an integer parameter") + + .def( + "add_real", + [](parthenon::ParameterInput &self, const std::string &block, + const std::string &name, + parthenon::Real value) { self.AddParsedParameter(block, name, value); }, + "Add a real parameter") + + .def( + "add_bool", + [](parthenon::ParameterInput &self, const std::string &block, + const std::string &name, + bool value) { self.AddParsedParameter(block, name, value); }, + "Add a boolean parameter") + + .def( + "add_string", + [](parthenon::ParameterInput &self, const std::string &block, + const std::string &name, + const std::string &value) { self.AddParsedParameter(block, name, value); }, + "Add a string parameter") + + // Vector add methods + .def( + "add_int_vector", + [](parthenon::ParameterInput &self, const std::string &block, + const std::string &name, const std::vector &value) { + self.AddParsedParameter(block, name, value); + }, + "Add an integer vector parameter") + + .def( + "add_real_vector", + [](parthenon::ParameterInput &self, const std::string &block, + const std::string &name, const std::vector &value) { + self.AddParsedParameter(block, name, value); + }, + "Add a real vector parameter") + + .def( + "add_bool_vector", + [](parthenon::ParameterInput &self, const std::string &block, + const std::string &name, const std::vector &value) { + self.AddParsedParameter(block, name, value); + }, + "Add a boolean vector parameter") + + .def( + "add_string_vector", + [](parthenon::ParameterInput &self, const std::string &block, + const std::string &name, const std::vector &value) { + self.AddParsedParameter(block, name, value); + }, + "Add a string vector parameter") + + // Query methods (const, safe to call during parsing) + // Note: Get methods are intentionally NOT exposed to prevent premature + // finalization. Python input scripts should only ADD parameters, not query their + // values. + .def("does_parameter_exist", &parthenon::ParameterInput::DoesParameterExist, + "Check if a parameter exists") + + .def("does_block_exist", &parthenon::ParameterInput::DoesBlockExist, + "Check if a block exists") + + .def("get_parameter_names", &parthenon::ParameterInput::GetParameterNames, + "Get all parameter names in a block") + + .def("get_blocks_with_prefix", &parthenon::ParameterInput::GetBlockNamesWithPrefix, + "Get all blocks with a given prefix"); + + // NOTE: get_parameter_input() removed in favor of explicit parameter passing + // Python input files should define: def parthenon_init_parameters(pin): +} diff --git a/tmp/python_input_pr_description.md b/tmp/python_input_pr_description.md new file mode 100644 index 0000000000000..6a69f7d695695 --- /dev/null +++ b/tmp/python_input_pr_description.md @@ -0,0 +1,244 @@ +## PR Summary + +## Summary + +This PR adds support for Python scripts as input files (`.py` instead of `.pin`), building on the parser separation infrastructure from #1385. Python input files enable programmatic parameter generation with native command-line argument parsing, loops, conditionals, and integration with external data sources. This is obviously inspired by the recent features added to Riot, so we probably want to think about if/how these two things fit together. + +**Primary Use Case**: Python's argparse for rich command-line interfaces with type validation, choices, help text, and custom constraints. + +**Philosophy**: Minimal C++ core (~250 lines) that embeds Python and exposes `ParameterInput` methods. Optional Python helper classes (~420 lines) demonstrate usage but can be replaced with user-specific abstractions. + +## Key Features + +### 1. Python Command Line Arguments + +```bash +./app -i input.py --ndim=3 --nx=128 --cfl=0.3 --help +``` + +Python scripts can use argparse for validation: +- Type checking: `--nx=128` (enforces int) +- Help text: `--help` shows all available options +- Defaults: fallback values if not specified +- Complex validation: parameter interdependencies + +### 2. Programmatic Configuration + +```python +import argparse + +def parthenon_init_parameters(pin): + """Configure parameters based on command line arguments.""" + parser = argparse.ArgumentParser() + parser.add_argument("--ndim", type=int, choices=[1,2,3], default=2) + parser.add_argument("--nx", type=int, default=64) + args, _ = parser.parse_known_args() + + pin.add_int("parthenon/mesh", "nx1", args.nx) + pin.add_int("parthenon/mesh", "nx2", args.nx if args.ndim >= 2 else 1) + pin.add_int("parthenon/mesh", "nx3", args.nx if args.ndim >= 3 else 1) +``` + +### 3. Flexible Abstractions + +Users can choose their approach: +- **Direct API**: Call `pi.add_int()`, `pi.add_real()`, etc. directly +- **Helper classes**: Use provided `InputFile`/`Block` classes (optional) +- **Custom parsers**: Write JSON/YAML parsers that populate ParameterInput +- **Application-specific**: Build domain-specific parameter generators + +## Implementation + +### C++ Core (~250 lines) + +**New files**: +- `src/parameter_parsers/python_parser.{hpp,cpp}` (~120 lines) + - Embeds Python interpreter via pybind11 + - Executes user's `.py` script to load function definitions + - Calls `parthenon_init_parameters(pin)` function with ParameterInput object + - Returns populated ParameterInput to C++ + +- `src/pybind/parameter_input_bindings.cpp` (~135 lines) + - Exposes `add_int`, `add_real`, `add_bool`, `add_string`, `add_*_vector` methods + - Exposes query methods: `does_parameter_exist`, `get_parameter_names`, etc. + - **Intentionally does NOT expose `Get()` methods during parsing** (would trigger `FinalizeParsing()` and break command-line overrides) + +- `src/pybind/CMakeLists.txt` (~80 lines) + - Builds `parthenon.so` Python module + - Creates symlinks for single PYTHONPATH: `export PYTHONPATH=build/lib/python:$PYTHONPATH` + +**Modified files**: +- `src/parthenon_manager.cpp`: Detect `.py` extension and call `LoadParameterInputFromPython()` +- `src/config.hpp.in`: Add `PARTHENON_ENABLE_PYTHON_BINDINGS` define +- `src/CMakeLists.txt`: Add `parameter_parsers/*.{cpp,hpp}` to library + +### Python Tooling (~420 lines, optional) + +**New package**: `scripts/python/packages/parthenon_input/` +- `input_generator.py`: `InputFile` and `Block` classes for structured parameter building +- `__init__.py`: Exports and `mpi_print()` helper for rank 0 printing + +**Example**: `example/fine_advection/parthinput.advection.py` (151 lines) +- Demonstrates argparse for ndim-agnostic configuration +- Shows programmatic parameter generation +- Updated README with Python usage instructions + +### Documentation + +**New files**: +- `docs/python_input_summary.md` (~450 lines) + - Architecture overview (minimal core philosophy) + - Multiple usage patterns (direct API, helper classes, JSON, programmatic) + - Integration with parser separation refactor + - Extensibility examples + +- `src/pybind/README.md` (~100 lines) + - Python API reference + - PYTHONPATH setup instructions + - Example usage patterns + +## Integration with Core Refactor (#1385) + +Python input integrates seamlessly with parser separation: + +1. Python file executes to load function definitions +2. C++ calls `parthenon_init_parameters(pin)` which populates `ParameterInput` via `add_*()` methods (uses `AddParsedParameter()` interface) +3. Function returns, interpreter shuts down, populated `ParameterInput` returned to C++ +4. C++ applies command line overrides via `ModifyFromCmdline()` (Parthenon-style `block/param=value`) +5. C++ calls `FinalizeParsing()` to mark parsing complete +6. Application queries parameters via `Get()` and `GetOrAdd()` + +Python scripts run **before** `FinalizeParsing()`, using the same parser interface as text files. The explicit function call pattern (`parthenon_init_parameters(pin)`) provides a clear entry point with no magic global variables. + +## Build Requirements + +```cmake +-DPARTHENON_ENABLE_PYTHON_BINDINGS=ON # Enable Python input support +``` + +**Dependencies**: +- pybind11 (found via `find_package(pybind11)`) +- Python 3 development headers + +**Without Python support**: +- `.py` input files trigger clear error message: "Python input detected but not enabled at build time" +- No runtime Python dependency +- All code guarded by `#ifdef PARTHENON_ENABLE_PYTHON_BINDINGS` + +## Usage Example + +```bash +# Build with Python support +cmake -DPARTHENON_ENABLE_PYTHON_BINDINGS=ON .. +make + +# Set PYTHONPATH +export PYTHONPATH=/path/to/build/lib/python:$PYTHONPATH + +# Run with Python input +./fine_advection -i parthinput.advection.py --ndim=2 --nx=128 +``` + +## Command Line Argument Flow + +```bash +./app -i input.py --nx=128 parthenon/mesh/refinement=static +``` + +1. C++ executes Python file to load function definitions +2. C++ calls `parthenon_init_parameters(pin)` +3. Inside function, Python sees: `sys.argv = ["input.py", "--nx=128", "parthenon/mesh/refinement=static"]` +4. Python uses `parse_known_args()` to consume `--nx=128`, ignores rest +5. Function populates ParameterInput based on parsed arguments and returns +6. C++ receives populated ParameterInput +7. C++ applies `ModifyFromCmdline()` to override `parthenon/mesh/refinement=static` + +Both Python-style (`--flag=value`) and Parthenon-style (`block/param=value`) arguments work together. + +## Testing + +- **Unit tests**: Parameter input bindings tested via existing C++ test infrastructure +- **Regression test**: `example/fine_advection/parthinput.advection.py` demonstrates full workflow +- **Build configurations**: Both `PARTHENON_ENABLE_PYTHON_BINDINGS=ON` and `OFF` tested + +## Lines of Code + +| Component | Lines | Required | +|-----------|-------|----------| +| C++ core (parameter_parsers, pybind) | ~250 | Yes | +| Python tooling (parthenon_input package) | ~420 | No (example) | +| Documentation (docs, README) | ~550 | - | +| Example (parthinput.advection.py) | ~150 | No (example) | + +**Total new code**: ~1370 lines (~250 required, ~1120 optional/documentation) + +## Future Extensions + +The minimal core enables diverse use cases: + +**JSON/YAML input**: +```python +import json + +def parthenon_init_parameters(pin): + """Load configuration from JSON file.""" + config = json.load(open("config.json")) + for block, params in config.items(): + for key, value in params.items(): + if isinstance(value, int): + pin.add_int(block, key, value) + elif isinstance(value, float): + pin.add_real(block, key, value) + # ... etc +``` + +**Parameter sweeps**: +```python +import os + +def parthenon_init_parameters(pin): + """Configure parameters based on environment variable.""" + run_id = int(os.environ["RUN_ID"]) + pin.add_real("problem", "amplitude", 0.1 * run_id) +``` + +**Application-specific abstractions**: +```python +from my_app import Setup + +def parthenon_init_parameters(pin): + """Use application-specific configuration helper.""" + setup = Setup(param1="value", param2=1.4) + setup.configure_parameter_input(pin) # User's custom logic +``` + +## Breaking Changes + +None. This is a pure addition: +- Existing `.pin` files work unchanged +- Python support is opt-in at build time +- No changes to existing public APIs + +## Depends On + +- #1385 (Parser separation refactor) - must be merged first + + + +## PR Checklist + + + +- [x] Code passes cpplint +- [x] New features are documented. +- [ ] Adds a test for any bugs fixed. Adds tests for new features. +- [x] Code is formatted +- [x] Changes are summarized in CHANGELOG.md +- [ ] Change is breaking (API, behavior, ...) + - [ ] Change is *additionally* added to CHANGELOG.md in the breaking section + - [ ] PR is marked as breaking + - [ ] Short summary API changes at the top of the PR (plus optionally with an automated update/fix script) +- [ ] CI has been triggered on [Darwin](https://re-git.lanl.gov/eap-oss/parthenon/-/pipelines) for performance regression tests. +- [ ] Docs build +- [x] Any contribution that was created or modified with the assistance of generative AI must have a comment disclosing this such as `// This file was made in part with generative AI.` +- [x] (@lanl.gov employees) Update copyright on changed files