[DevOps]: Add Chrysalis ingestion wrapper and deploy - #169
[DevOps]: Add Chrysalis ingestion wrapper and deploy #169tomvothecoder wants to merge 20 commits into
Conversation
|
@tomvothecoder Once I am clear on the boundaries to the term "NERSC ingestion wrapper", I should be able to comprehent "Chrysalis ingestion wrapper". The term "scheduler-agnostic" refers to Jenkins? (I always considered cron to be universal...). |
TonyB9000
left a comment
TonyB9000
left a comment
There was a problem hiding this comment.
Configures a call to the hpc_archive_ingestor. Understandable.
What process sets "SIMBOARD_API_BASE_URL" and "SIMBOARD_API_TOKEN"?
|
Hmmmm. The "Tom Requested your review" took me to the page with 7 files to examine, each with a "submit-review" option. As soon as I completed the first one, all 7 vanished... |
Accidentally tagged you for review. I meant to assign this PR you. It is fixed now. |
|
@tomvothecoder "Accidentally tagged you for review". OK, (I think colleges should offer a master's program in github). |
|
Chrysalis and other non-NERSC sites require upload-based ingestion rather than path-based ingestion, so follow-up work is tracked in #207 for a state-first HPC upload flow with DB-backed dedupe parity. |
|
Using the "upload-based' vs "path-based" terminology, my thought was that when the NERSC upload-receiving system was deliverd an upload from a non-NERSC system, it could open it in the existing NERSC PA-directory under (say) "From_crysalis/<new_exec_ids>" and then process it with the existing "path-based" codes - assuming PACE would not interfere with it (and vice-versa). But on second thought, to avoid PACE crossing, it would be best to open it in a separate "PACE-unaware" directory. |
|
@tomvothecoder I am preparing to exercise "hpc_upload_archive_ingestor.py" on chrysalis, to see the logs and flow (in dry-run) in action, discover parameter faults, etc. QUESTION: Although, on NERSC, the backend ingestion is "path-based" (returnsp paths for ingestion), it could in principle run the "https-transfer-based" codes just as easily. I might try a dryrun on NESRC/Perlmutter first, since that configuration is already a known item. Then, differences in behavior on chrysalis would stand out. Does that make sense? |
22a1a88 to
feae197
Compare
|
@tomvothecoder Apologies if I'm doing this wrong. I attempted to test the "hpc_upload" on NESRC, thinking "--help" might be helpful. To get started, I needed an environment where I could install things, so: |
|
Hey Tony, happy to help and no apologies needed.
SimBoard defines the Python backend dependencies in You can run
I'd checkout this branch now that I've rebased it on the latest The |
|
@tomvothecoder I get the latest stuff - but I have made progress. My latest run_script (NERSC dry_run test) says: The output indicates that I am missing "archive_root” and “has_api_token”. By examining the "nersc" "_build_config" function, I can see what variables exist to push into the environment. I'll checkout branch #169 on both NERSC and Chrysalis to do comparisons in outputs. |
|
@tomvothecoder git gets me again: You wrote: "I'd checkout this branch now that I've rebased it on the latest main commit." is "this branch" off of main, as you had advised? Or is it off of a fork?? ((test_simboard) ) (base) [ac.bartoletti1@chrlogin1 simboard]$ git branch -a
When I get too confused, I do a clean "git clone". Then I can do one of these: Which is appropriate in this case? |
|
@tomvothecoder If I pull down a branch off of someone's fork, am I in that fork, or can I pull that into a new branch of my local main? The persistence of branches and forks, between local and remote, is a bit of a mystery. |
This branch ( Something like this (I did not verify correctness): git remote add tomvothecoder https://github.com/tomvothecoder/simboard
git checkout tomvothecoder feature/154-ingestion-ites I usually work directly on upstream and not fork when possible, but in this case I use a fork for separate testing purposes. |
|
@tomvothecoder Sorry, I guess I must pull from your fork. Quick test: I cd to ``/home/ac.bartoletti1/gitrepo/simboard/backend" and issue
The result: |
|
@tomvothecoder The line in "chrysalis.sh" script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; echo $script_dir clearly wont work for defining "backend_root" as backend_root="$(cd "${script_dir}/../../../.." && pwd)" I will modify chrysalis.sh to provide a "backend_root" that does not depend upon the user location., at least for test purposes. |
|
@tomvothecoder Works better now that it can find "backentd/apps" When I use this for "chrysalis.sh": and issue these exports: I get: I guess, even "dry_run" requires real URLs and API_tokens. That is because we need "state" up front. |
|
Hi @tomvothecoder The document also says: One-case-per-request rule:
The term "alongside the archive" is a bit ambiguous. Would this be accurate?
Or am I misunderstanding the intent? |
Great to see the progress! Yes, the dry run needs to query the SimBoard database via the REST API. I will send the
Your info sounds more accurate, thanks for the suggestion. Can you point me to the source document with this info? I will update it. |
|
@tomvothecoder Running "chrysalis.sh" with the full (DRY_RUN) parameters yieded the following summary (folded for readability): Questions that arise:
Observation: The bulk of work getting to this point involved stuffing the right ENV VARS and having created an environment where misc modules like "dateutils" could be installed. On Chrysalis, I performed On NERSC/Perlmutter, I simply replaced "python3.12" with "python3.11". I intend to perform the same test on Perlmutter, just to exercise the mechanisms of networking. |
|
@tomvothecoder For comparison, running the equivalent commands on perlmutter (swapping our parameters where necessary), we obtain the summary: I suppose I should re-run the chrysalis test, using "OLD_PERF" as the root_PA directory. It is HUGE. |
Since this is a first-time dry-run on the Chrysalis
That exact counter is not in So yes: a test expecting that exact field is probably unrealistic or stale.
The ingestor script only asks SimBoard for enough existing ingestion state to decide which archive cases and their executions are new and may be candidates for ingestion. It does not fetch, return, or summarize the full database state. It also does not show the database query because the query is behind the SimBoard API, not inside the ingestor script. So this is not a Chrysalis-specific DB query in the ingestor. It is an API request filtered by the configured If more detail is needed, the API response or ingestor summary would need to be expanded to include counts like total known cases, known execution IDs, skipped known cases, and machine filter used. Happy for you to open a new GitHub issue to expand logging in https://github.com/E3SM-Project/simboard/blob/main/backend/app/scripts/ingestion/nersc_archive_ingestor.py and https://github.com/E3SM-Project/simboard/blob/main/backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py.
They are dry-run logging counters.
The point is to avoid massive logs when many candidate cases are found. It does not change which cases are candidates or which cases would be ingested. |
I don't think this is going to work yet as the directory structure of "OLD_PERF" is different from "performance_archive". We might also want to be targeted in what we ingest from "OLD_PERF". This will require guidance Rob/Jill. |
1cef6f7 to
3bc4d45
Compare
|
My reply to your comment:
Can you update the existing nersc.crontab.example? Afterwards, can you document (very simply, step-by-step) the setup process for all of the operations scripts and crontab here? I'll use AI to review. |
|
Also make sure to pull the latest commit. I fixed rebase conflicts. |
|
@tomvothecoder I pulled the latest commits to your fork/branch, and confirmed that all of my changes are included. As far as "the setup process for all of the operations scripts and crontab here", I'm not sure if you want me to update the README, or to establish a separate "Setup for Scripts and Crontab" document. I will assume the latter. |
|
@tomvothecoder I added a tentative markdown doc to the "sites" directory, detailing setup instructions. It now occurs to me that the site-config file has lines that are NOT site-specific, and should be maintained in the launch script itself. At least one of these (the module-path to the site launch script) should be eliminated from the site config and coded into the script. |
Actually keeping that information in the README makes sense. We can update the top-level documentation (the one I originally linked) to that README instead. |
| # I DON'T KNOW ABOUT THESE | ||
| # Diagnostics provenance scan: start dry-run, inspect logs, then set DRY_RUN=false. | ||
| # 20 * * * * cd ${REPO_DIR} && ${REPO_DIR}/backend/app/scripts/ingestion/sites/nersc-diagnostics-scanner.sh >> ${REPO_DIR}/backend/app/scripts/ingestion/sites/nersc-diagnostics-scanner.log 2>&1 | ||
|
|
||
| # Chrysalis diagnostics provenance scan; use its local checkout path for REPO_DIR. | ||
| # 25 * * * * cd ${REPO_DIR} && MACHINE_NAME=chrysalis ${REPO_DIR}/backend/app/scripts/ingestion/sites/lcrc-diagnostics-scanner.sh >> ${REPO_DIR}/backend/app/scripts/ingestion/sites/lcrc-diagnostics-scanner.log 2>&1 |
There was a problem hiding this comment.
This is a separate cronjob that I implemented in #293. I think it should be a single line rather than two and renamed to diagnostics-scanner.sh. I'll fix this in another pull request.
|
@tomvothecoder I was testing last night and this morning, and failing because my freshly-cloned simboard repo no longer had a "backend/.venv". The site launch script simply (and silently) fails at that point because the original test exits when ".venv" is absent, before it gets a chance to test for "-x executable". I just pushed an update to correct this. I plan to run a "MAX_CASES_PER_RUN=1" with DRY_RUN=false. We should have results in time for today's sync. |
|
@tomvothecoder The configuration output (job is still running): |
|
@tomvothecoder and the tail of the output: (I have a hard time distinguishing at what point in the output we have transitioned beyond "dry_run" to packaging and transmission. These phases should perhaps be made more clearly distinguished). The full output log is available for examination here:
|
I think the site launch script should have a check to see if the The |
|
@tomvothecoder "suggest not deleting entire repos ". Yes, I understand. Still, some user may do this, and the "existence" test should not fail to output a message in that case. See you soon! |
It looks like your single case upload worked: https://simboard-dev.e3sm.org/cases/79a4e11e-9d09-4c36-af08-61e07634d051 |
|
Replying to your comment quoted below from another GitHub issue.
The run completed most of the expected work, but 11 case submissions failed and still need investigation. The field definitions are documented in the SimBoard Runner counter and log field terms.
Chrysalis catalog checkThe website and database both show 791 Chrysalis cases.
The 39 pre-existing cases include 37 created on August 24, consistent with the v3 ingestion, and 2 created before the general run on August 26. The 934 records are upload requests, not Case records; multiple uploads can add executions to the same Case. The SQL window is not an exact run-level match: it contains three more records than the runner’s 931 successful HTTP submissions. One item remains to reconcile: Next steps
SQL query: Chrysalis catalog reconciliationWITH run_parameters AS (
SELECT
(TIMESTAMP '2026-08-26 16:01:32.774'
AT TIME ZONE 'America/Chicago')
- INTERVAL '7883.398 seconds' AS started_at,
TIMESTAMP '2026-08-26 16:01:32.774'
AT TIME ZONE 'America/Chicago' AS finished_at
),
chrysalis AS (
SELECT id FROM machines WHERE LOWER(name) = 'chrysalis'
),
case_counts AS (
SELECT
COUNT(*) AS chrysalis_cases_currently,
COUNT(*) FILTER (WHERE c.created_at < p.started_at)
AS cases_existing_before_run,
COUNT(*) FILTER (
WHERE c.created_at >= p.started_at
AND c.created_at < p.finished_at
) AS cases_created_during_run,
COUNT(*) FILTER (WHERE c.created_at >= p.finished_at)
AS cases_created_after_run,
COUNT(*) FILTER (
WHERE c.created_at < p.started_at
AND (c.created_at AT TIME ZONE 'America/Chicago')::date
= DATE '2026-08-24'
) AS cases_created_august_24,
COUNT(*) FILTER (
WHERE c.created_at < p.started_at
AND (c.created_at AT TIME ZONE 'America/Chicago')::date
= DATE '2026-08-26'
) AS cases_created_august_26_before_run
FROM cases c
JOIN chrysalis m ON m.id = c.machine_id
CROSS JOIN run_parameters p
),
run_ingestions AS (
SELECT i.*
FROM ingestions i
JOIN chrysalis m ON m.id = i.machine_id
CROSS JOIN run_parameters p
WHERE i.source_type = 'hpc_upload'
AND i.created_at >= p.started_at
AND i.created_at < p.finished_at
),
ingestion_counts AS (
SELECT
COUNT(*) AS ingestion_records,
COUNT(DISTINCT source_reference) AS uploaded_case_paths,
COALESCE(SUM(created_count), 0) AS executions_created,
COALESCE(SUM(duplicate_count), 0) AS executions_skipped_as_duplicates,
COALESCE(SUM(error_count), 0) AS ingestion_errors,
COUNT(*) FILTER (WHERE status = 'failed') AS failed_ingestion_records,
COALESCE(SUM(
CASE WHEN status = 'failed'
THEN COALESCE(jsonb_array_length(processed_execution_ids), 0)
ELSE 0 END
), 0) AS failed_selected_execution_ids
FROM run_ingestions
),
execution_counts AS (
SELECT COUNT(DISTINCT e.case_id) AS chrysalis_cases_represented
FROM executions e
JOIN run_ingestions i ON i.id = e.ingestion_id
)
SELECT cc.*, ic.*, ec.*
FROM case_counts cc
CROSS JOIN ingestion_counts ic
CROSS JOIN execution_counts ec; |
|
@TonyB9000 Can you provide the log file, either attached here or the path on LCRC? |
|
@tomvothecoder The paths to each of these log files is [SIMBOARD_ROOT/operations/]. The one BIG logfile is /lcrc/group/e3sm2/simboard/operations/SBCS-20260826_185001.log (Note: I have temporarily suspended all cronjobs). If I list the contents of operations by last modification date (ls -lrt), you can see that after a while, it takes about 20 minutes to complete a "nothing to add" run, so we see alternation between large and very small logfiles (small being aborted runs). But after a while I see several large logs listed in sequence - not sure why. Output of ls -lrt: The three "subsequent" additions may have appeared overnight, and been processed by later collection runs. The 11 "failed" and other such items should be derivable by filtering the log file appropriately, and extracting both CASE and EXEC_ID for each. If that is not easy to do filtering the logs, we need to revise the structure of the log files. I can look for "failed" and see what turns up and find other things to filter upon (e.g. status_code, error, and archive_bytes). Likewise, we really need the ability to provide (say) a file containing a list of CASE/EXEC_ID pairs that can be used to isolate and ingest (or assess) exactly those runs. Short of this, I don't know quite how to debug these failed case/execs. |
|
@tomvothecoder Let me try this again:
result: |
|
I'm seeing "413 Request Entity Too Large" and "503 Service Temporarily Unavailable". |
|
@tomvothecoder These 6 seemed to trigger the "too large" error: and these 3 triggered the "503 service unavailable": The last two are |
|
@TonyB9000 Good news, the log contains the information needed to classify all 11 failed submissions and reconcile their selected execution IDs. I will let you know when to invoke another ingestion job manually to see if these get picked up. Request-level failure breakdownTotal: The 11 failed submissions contained 321 selected execution IDs. NGINX
|
| Case | Selected IDs | Archive size |
|---|---|---|
v2.LR.BGC-LNDATM.FireUpdate |
41 | 59.60 MiB |
20250401...n0030 |
1 | 54.56 MiB |
20250401...n0030_31_60 |
1 | 56.51 MiB |
runoffSpreading150e300 |
62 | 161.98 MiB |
runoffSpreading50e100 |
61 | 160.41 MiB |
runoffSpreadingBaseline |
62 | 159.77 MiB |
Root cause: All six archives exceed the current 52m upload limit.
- Next step: Increase both NGINX and backend upload limits, then retry these cases. -- Done in fix: raise archive upload limits #308
URL error: [Errno 104] Connection reset by peer
| Case | Selected IDs | Archive size |
|---|---|---|
20251107.v3.HR.piControl-Spinup |
84 | 300.95 MiB |
Root cause: The connection was reset; this archive is also much larger than the current upload limits.
- Next step: Increase both limits, then retry while monitoring the upload. -- Done in fix: raise archive upload limits #308
NGINX 503 Service Temporarily Unavailable
| Case | Selected IDs | Archive size |
|---|---|---|
v3.LR.ssp370_0261 |
2 | 8.98 MiB |
20251203...chrysalis |
4 | 4.78 MiB |
20251208...chrysalis |
2 | 0.66 MiB |
Root cause: NGINX could not reach a ready backend.
- Next step: Confirm backend availability, then retry these cases. -- The backend is up, the next ingestion job should pick these up automatically (if not already). Need to verify if it's been ingested.
psycopg.errors.UniqueViolation
| Case | Selected IDs | Archive size |
|---|---|---|
20251209...gis4to40... |
1 | 0.33 MiB |
Root cause: Another request created the same Case identity concurrently.
- Next step: Retry the case; it may now succeed because the Case exists. -- This should be done on the next ingestion job (if not already). Need to verify if it's been ingested.
I am ready for you to run another Chrysalis staging ingestion job (if cron is not setup right now). Then we can see if those cases are still failing to upload. |
2 participants
Description
This adds a scheduler-agnostic HPC archive ingestor entrypoint and a thin Chrysalis site wrapper for existing Jenkins-driven metadata ingestion.
hpc_archive_ingestormodule that delegates to existing NERSC ingestorsites/chrysalis.shwrapper with Chrysalis archive and state defaults plus required API env varsTask
Finish turning this branch into a deployable ingestion path for Chrysalis first, then use the same pattern for other sites once access is available.
SIMBOARD_API_BASE_URLandSIMBOARD_API_TOKENshould be stored and injected in the Chrysalis Jenkins job.Checklist
Deployment Notes (if any)
No special deployment steps.
F