Skip to content

Fix: storage_utils: Check multipath devices on all nodes - #2173

Open
liangxin1300 wants to merge 2 commits into
ClusterLabs:masterfrom
liangxin1300:20260817_multipath_multinode
Open

Fix: storage_utils: Check multipath devices on all nodes#2173
liangxin1300 wants to merge 2 commits into
ClusterLabs:masterfrom
liangxin1300:20260817_multipath_multinode

Conversation

@liangxin1300

Copy link
Copy Markdown
Collaborator

Multipath validation for SBD and cluster filesystems only checked the local node. This change runs the multipath inspection on each target node and reports where the invalid device was found.

  • Add peer-aware MultipathInspector checks
  • Pass node lists from SBD and cluster FS validation
  • Log multipath errors with the failing node
  • Add unit tests for multi-node checks

@liangxin1300
liangxin1300 force-pushed the 20260817_multipath_multinode branch 2 times, most recently from 2ba533a to fcba122 Compare August 17, 2026 02:58
@codecov

codecov Bot commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 70.70%. Comparing base (a973bf9) to head (39045e5).

Files with missing lines Patch % Lines
crmsh/cluster_fs.py 0.00% 1 Missing ⚠️
Additional details and impacted files
Flag Coverage Δ
integration 55.34% <10.00%> (-0.02%) ⬇️
unit 52.87% <95.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
crmsh/sbd.py 77.78% <100.00%> (ø)
crmsh/storage_utils.py 98.59% <100.00%> (+0.07%) ⬆️
crmsh/cluster_fs.py 80.80% <0.00%> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@liangxin1300
liangxin1300 marked this pull request as ready for review August 17, 2026 05:46
Comment thread crmsh/storage_utils.py Outdated
for node in node_list or [utils.this_node()]:
inspector = cls(dev, node)
if inspector._is_under_multipath():
error_msg = f"Device {dev} is under multipath on {node}, please provide the multipath device instead"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please provide the multipath device instead

I am not sure if this suggestion is accurate. If a device is only under multipath on some of the nodes, it is probably a different problem, for example, inconsistent device file names in different node. I incline to report only the error without providing suggestions blindly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inconsistent device file names in different node

This is the prerequisite to configuring disk-based SBD, see https://documentation.suse.com/sle-ha/15-SP7/html/SLE-HA-all/cha-ha-storage-protect.html#sec-ha-storage-protect-req

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even though consistent device naming is a prerequisite, validation logic should detect and report when this prerequisite is unsatisfied rather than assuming it is met. Suggesting users provide the multipath device when names are actually inconsistent is misleading and should be avoided.

@liangxin1300
liangxin1300 force-pushed the 20260817_multipath_multinode branch 2 times, most recently from 0a335fe to ea455cf Compare August 19, 2026 08:17
Comment thread crmsh/storage_utils.py Outdated
Comment on lines +212 to +214
resolved = Path(dev).resolve()
cmd = f"lsblk -dn -o PKNAME {shlex.quote(str(resolved))}"
_, out, _ = self._shell.get_rc_stdout_stderr_without_input(None, cmd)
_, out, _ = self._shell.get_rc_stdout_stderr_without_input(self._peer, cmd)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks problematic to resolve symlink on localhost and run lsblk on a remote host.

Moreover, PKNAME returns the kernel device name, which will possibly negate the benefits of using stable names like /dev/disk/by-id/.... I don't know what kinds of symlink you have tested here. It looks quite complex when combining iscsi, device mapper, multipath and partitions together.

Multipath validation for SBD and cluster filesystems only checked the local node. This change runs the multipath inspection on each target node and reports where the invalid device was found.

- Add peer-aware MultipathInspector checks
- Pass node lists from SBD and cluster FS validation
- Log multipath errors with the failing node
- Add unit tests for multi-node checks
MultipathInspector can inspect devices on remote cluster nodes. Resolving device symlinks locally can produce paths that do not exist or have different meaning on the target node, causing incorrect multipath validation.

- Query lsblk on the target node
- Preserve the original device path for lsblk
- Parse PKNAME/KNAME pair output
- Use KNAME for whole-disk devices
- Raise an error when no kernel name is found
- Cover remote-safe and failure cases in tests
@liangxin1300
liangxin1300 force-pushed the 20260817_multipath_multinode branch from ea455cf to 39045e5 Compare August 26, 2026 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants