Skip to content

Make setDVsFromHistory parallel safe - #489

Open
timryanb wants to merge 3 commits into
mdolab:mainfrom
timryanb:setdvs-from-history-parallel-safe
Open

Make setDVsFromHistory parallel safe#489
timryanb wants to merge 3 commits into
mdolab:mainfrom
timryanb:setdvs-from-history-parallel-safe

Conversation

@timryanb

@timryanb timryanb commented Aug 25, 2026

Copy link
Copy Markdown

Purpose

Modify setDVsFromHistory method to read the history file on the root proc only.

The method opened the SQLite-backed history file (via SqliteDict) on every processor that executed it. SQLite takes POSIX fcntl() locks, so on network filesystems (e.g. NFS scratch on HPC clusters) this meant every rank taking file locks on the same file through the NFS lock manager, which can bottleneck and hang the whole job before the optimizer starts.

Now only the root processor opens the history file and the resulting DVs (or any exception, to avoid deadlocking the collective) are broadcast to all processors, mirroring the rank-0 guard already used in Optimizer._setHistory.

Expected time until merged

1 week

Type of change

run optProb.setDVsFromHistory on any sqlite history file in parallel.

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (non-backwards-compatible fix or feature)
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no API changes)
  • Documentation update
  • Maintenance update
  • Other (please describe)

Testing

Checklist

  • I have run ruff check and ruff format to make sure the Python code adheres to PEP-8 and is consistently formatted
  • I have formatted the Fortran code with fprettify or C/C++ code with clang-format as applicable
  • I have run unit and regression tests which pass locally with my changes
  • I have added new tests that prove my fix is effective or that my feature works
  • I have added necessary documentation

…oc only

The method opened the SQLite-backed history file (via SqliteDict) on
every processor that executed it. SQLite takes POSIX fcntl() locks, so
on network filesystems (e.g. NFS scratch on HPC clusters) this meant
every rank taking file locks on the same file through the NFS lock
manager, which can block indefinitely and hang the whole job before
the optimizer starts.

Now only the root processor opens the history file and the resulting
DVs (or any exception, to avoid deadlocking the collective) are
broadcast to all processors, mirroring the rank-0 guard already used
in Optimizer._setHistory.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@timryanb
timryanb requested a review from marcomangano as a code owner August 25, 2026 16:27
@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.22%. Comparing base (9ee4f75) to head (f255e38).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #489   +/-   ##
=======================================
  Coverage   54.22%   54.22%           
=======================================
  Files           1        1           
  Lines         225      225           
=======================================
  Hits          122      122           
  Misses        103      103           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant