Skip to content

Fix problems of softdog watchdog driver - #2155

Open
liangxin1300 wants to merge 4 commits into
ClusterLabs:masterfrom
liangxin1300:20260708_watchdog_issue
Open

Fix problems of softdog watchdog driver#2155
liangxin1300 wants to merge 4 commits into
ClusterLabs:masterfrom
liangxin1300:20260708_watchdog_issue

Conversation

@liangxin1300

@liangxin1300 liangxin1300 commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

Problems:

  1. Peer node doesn't load the softdog watchdog driver when running the sbd stage
    see:
  2. crm sbd status shows <unknown> instead of softdog when it's configured
  3. crm cluster health sbd should give a warning when using softdog as watchdog driver

Changes:

  1. Fix: watchdog: Load watchdog driver correctly
    • When running in sbd stage, load watchdog driver on all nodes
    • When running in join process, load driver on local node only
  2. Dev: watchdog: Resolve from configured driver
    • Avoid duplicated query-watchdog parsing
    • Reuse Watchdog.QUERY_CMD in ui_sbd
    • Share parser for local and remote checks
    • Keep when configured driver is unloaded
    • Cover parser behavior with unit tests
  3. Dev: watchdog: Warn when SBD uses softdog
    • Add public softdog usage warning helper
    • Reuse query-watchdog parser for detection
    • Run warning during SBD config checks
    • Cover warning paths with unit tests
  4. Dev: watchdog: Prefer hardware watchdog by default
    • Remove duplicate verify path
    • Drop unused first-device helper
    • Prefer non-softdog query results
    • Fall back to softdog when no hardware device is known
    • Update watchdog unit tests

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.27273% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.58%. Comparing base (157564b) to head (18e8d20).

Files with missing lines Patch % Lines
crmsh/watchdog.py 85.41% 7 Missing ⚠️
Additional details and impacted files
Flag Coverage Δ
integration 55.21% <21.81%> (-0.03%) ⬇️
unit 52.74% <87.27%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
crmsh/sbd.py 77.78% <100.00%> (+0.02%) ⬆️
crmsh/ui_sbd.py 83.93% <100.00%> (ø)
crmsh/utils.py 64.88% <100.00%> (ø)
crmsh/watchdog.py 92.18% <85.41%> (-4.90%) ⬇️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@liangxin1300
liangxin1300 force-pushed the 20260708_watchdog_issue branch from 04391f6 to 20769ca Compare July 13, 2026 07:23
@liangxin1300 liangxin1300 changed the title Improve sbd watchdog Fix problems of softdog watchdog driver Jul 13, 2026
@liangxin1300
liangxin1300 force-pushed the 20260708_watchdog_issue branch from 20769ca to 06e0df9 Compare July 13, 2026 13:33
@liangxin1300
liangxin1300 marked this pull request as ready for review July 13, 2026 14:10
@liangxin1300
liangxin1300 marked this pull request as draft July 13, 2026 14:38
@liangxin1300
liangxin1300 force-pushed the 20260708_watchdog_issue branch 2 times, most recently from 53458db to f4c7055 Compare July 25, 2026 12:31
@liangxin1300
liangxin1300 marked this pull request as ready for review July 25, 2026 13:14
- When running in sbd stage, load watchdog driver on all nodes
- When running in join process, load driver on local node only
- Avoid duplicated query-watchdog parsing
- Reuse Watchdog.QUERY_CMD in ui_sbd
- Share parser for local and remote checks
- Keep <unknown> when configured driver is unloaded
- Cover parser behavior with unit tests
- Add public softdog usage warning helper
- Reuse query-watchdog parser for detection
- Run warning during SBD config checks
- Cover warning paths with unit tests
- Remove duplicate verify path
- Drop unused first-device helper
- Prefer non-softdog query results
- Update watchdog unit tests
@liangxin1300
liangxin1300 force-pushed the 20260708_watchdog_issue branch from f4c7055 to 18e8d20 Compare July 31, 2026 02:16
Comment thread crmsh/watchdog.py

@staticmethod
def _load_watchdog_driver(driver):
def _load_watchdog_driver(driver, join=False):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing join here couples a low level driver loading util with the cluster-level concept (init vs join). A better approach is to pass node_list here and let the caller decide.

Comment thread crmsh/watchdog.py
if sbd_only and not re.search(r"Busy: .*sbd", identity):
continue
if driver == "<unknown>":
configured_driver = cls._get_configured_watchdog_driver()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is incorrect as get_watchdog_info accepts arbitrary out, possibly from a remote node, but _get_configured_watchdog_driver() query the local node for driver information.

Probably we should fix sbd to make it to print the correct driver instead of add workarounds in crmsh.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants