[WIP] Support AmSC IRI API

[EZoni]

Overview

Replaces #428 with further fixes and features.

Merge first

• [WIP] Harden training script: credentials validation, error handling, security fixes #432
• Improve style of parameters card #440
• Improve style of optimization card #441
• Simplify code in ModelManager class #442
• Improve style of models card #443
• Read password from stdin for podman-hpc login #444
• Improve style of Superfacility API card #445
• Rename Perlmutter variables specific to Superfacility API #454
• Update dashboard environment lock file #455
• Add AmSC Python client to dashboard environment #456
• Add globus-sdk to dashboard environment for Globus authentication #462

To do

• Update conda environment
• Add new GUI components
• Implement API connection
• Review and reduce code
• Split features and bug fixes

Notes

Here's a list of related repository, issues, tutorials, etc.:

• https://gitlab.com/amsc2/infrastructure-and-services/amsc-interfaces/amsc-python-client
• https://gitlab.com/amsc2/infrastructure-and-services/amsc-interfaces/amsc-python-client/-/work_items/3
• https://gitlab.com/amsc2/infrastructure-and-services/amsc-interfaces/amsc-python-client/-/work_items/7
• https://github.com/amsc-interfaces/amsc-client-tutorial/blob/main/notebooks/nersc_facility_tutorial.ipynb
• https://gitlab.com/amsc2/ai-services/intelligent-interfaces/agent-skills/iri-api/-/blob/main/SKILL.md (?)

[EZoni]

I confirm that, as of caefcc7, as expected I encounter the following error when I try to train an ML model with the AmSC IRI API backend:

Unable to complete remote training
Error occurred when executing remote training: 'FilesystemClient' object has no attribute 'upload'

[EZoni]

For easier debugging later on, a1283b7 removed the installation of the AmSC Python client from the local source expected to contain the original upload patch.

[EZoni]

I confirm that, as of caefcc7, as expected I encounter the following error when I try to train an ML model with the AmSC IRI API backend:

Unable to complete remote training
Error occurred when executing remote training: 'FilesystemClient' object has no attribute 'upload'

After some debugging, I think the amsc-client version with the latest upload fix has not yet been published as a package: https://gitlab.com/amsc2/infrastructure-and-services/amsc-interfaces/amsc-python-client/-/packages.

[EZoni]

After some debugging, I think the amsc-client version with the latest upload fix has not yet been published as a package: https://gitlab.com/amsc2/infrastructure-and-services/amsc-interfaces/amsc-python-client/-/packages.

When I try to bypass the package issue described in #439 (comment) by installing from source the latest version of amsc-client, I run into issues with the dependency iri-api-autogen, which does not seem to be installed automatically.

[EZoni]

Here's the latest version of the script I use to authenticate through Globus:

from amsc_client import Client


# Create client instance
client = Client(auth_method="globus")

# Check facility resources
nersc = client.facility("nersc")
for r in nersc.resources():
    print(r.name, r.status)

# Submit a job
perlmutter = nersc.resource("compute")
run_dir = "/pscratch/sd/e/ezoni/runs/iri"  # FIXME
job = perlmutter.submit(
    executable="/bin/echo",
    arguments=["Hello World!"],
    directory=run_dir,
    name="hello",
    queue="debug",
    account="m558",
    duration=300,
    nodes=1,
)
job.wait()

# Read job output via the filesystem API
home = nersc.resource("homes")
task = home.fs.head(f"{run_dir}/hello.stdout", lines=10)
task.wait()
print(task.result)

I think this should save a credentials.json in ~/.amsc automatically.