[1.7.1][Bug] Btrfs COW amplification exhausts transaction reserve, forces Main Storage read-only and stops Docker

# Bug report: Btrfs transaction-reserve exhaustion forces Main Storage read-only and stops Docker

## Summary

On a ZimaCube running ZimaOS v1.7.1, the Btrfs filesystem on the healthy Main Storage RAID repeatedly aborts a transaction with `errno=-28 (No space left)` and forces the filesystem read-only, even though multiple terabytes remain available.

Once Main Storage becomes read-only, Docker can no longer write its volume/network metadata. The Docker daemon exits and every container becomes unavailable. Rebooting temporarily restores service, but the same failure returns later.

The kernel trace and Btrfs reservation state closely match the upstream issue described as **COW amplification during `btrfs_search_slot()`**, where the transaction block reserve can be exhausted under memory/writeback pressure despite ample unallocated space:

https://www.spinics.net/lists/linux-btrfs/msg162178.html

This has now happened at least twice with the same kernel signature.

---

## Severity

**Critical / data-service outage**

- Main Storage is forced read-only.
- Docker fails completely.
- All containers and container-backed websites/services become unavailable.
- A reboot is required to restore operation temporarily.
- The issue is recurring.

No filesystem repair or remount was attempted after the abort; the system was left read-only until a controlled reboot.

---

## Environment

- **Hardware:** ZimaCube
- **OS:** ZimaOS v1.7.1
- **Build date:** 2026-08-21 08:17:47 UTC
- **Build classification from `/etc/os-release`:** `development`
- **Kernel:** `Linux 6.18.9 #4 PREEMPT(voluntary)`
- **Architecture:** amd64
- **Storage:** mdadm RAID5 (`md0`) with 7 active members
- **Filesystem:** Btrfs on `/dev/md0`
- **Mount point:** `/media/Main-Storage`
- **Docker data root:** `/media/Main-Storage/docker`
- **Workload:** multiple Docker containers, including PostgreSQL databases and browser-based services

Relevant `/etc/os-release` values:

```text
NAME="ZimaOS"
BOARD="ZimaCube"
VERSION="v1.7.1"
VERSION_ID=1.7.1
PRETTY_NAME="ZimaOS v1.7.1"
CPE_NAME=cpe:2.3:o:icewhale:zimaos:1.7.1:*:development:*:*:*:zimacube:*
BUILD_DATE="2026/08/21 08:17:47 UTC"
```

---

## Recurrence

The same failure was found in persistent kernel journals on two separate boots:

1. **2026-09-14 00:01:03 CEST**
2. **2026-09-21 07:13:46 CEST**

Both events show:

- `postgres` as the process performing the final metadata operation;
- warning at `fs/btrfs/delayed-inode.c:1572`;
- `btrfs_insert_delayed_dir_index()`;
- `btrfs_add_link()` and `btrfs_rename()`;
- transaction abort with error `-28`;
- empty `trans_block_rsv`;
- fully reserved `global_block_rsv`;
- metadata space still reported as not full;
- filesystem forced read-only.

PostgreSQL appears to be the final triggering writer, not necessarily the root cause. The upstream report demonstrates the same failure from ordinary file creation/rename under memory/writeback pressure.

---

## Actual result

Btrfs aborts the current transaction and changes Main Storage to read-only:

```text
BTRFS: Transaction aborted (error -28)
BTRFS: error (device md0 state A) in btrfs_add_link:6752: errno=-28 No space left
BTRFS info (device md0 state EA): forced readonly
BTRFS: error (device md0 state EA) in btrfs_rename:8624: errno=-28 No space left
```

Docker then fails because its data root is on that filesystem:

```text
failed to start daemon: error while opening volume store metadata database
(/media/Main-Storage/docker/volumes/metadata.db):
attempt to write a readonly database
```

All published container ports stop responding, while the ZimaOS host and its non-Docker web interface remain reachable.

---

## Expected result

- Btrfs must not report ENOSPC or abort a transaction while substantial data and metadata allocation capacity remains.
- Temporary transaction-reserve exhaustion must not force the entire filesystem read-only under a normal container workload.
- If this is a known kernel defect, ZimaOS should ship a kernel containing the applicable upstream fix/backport.
- ZimaOS should alert clearly when Main Storage is remounted read-only instead of only showing all applications as stopped.

---

## Evidence that the array/filesystem was not actually full

At the time of diagnosis after the failure:

```text
Filesystem      Size  Used Avail Use%
/dev/md0        6.0T  1.8T  3.8T  33%
```

Btrfs allocation summary:

```text
Device size:                 5.46 TiB
Device allocated:            1.72 TiB
Device unallocated:          4.10 TiB
Data, single:                1.70 TiB
Metadata, DUP:               6.00 GiB
Metadata used:               approximately 5.72 GiB
```

RAID state:

```text
md0 : active raid5 ...
      [7/7] [UUUUUUU]
```

Therefore, this is not conventional data-space exhaustion and not a degraded RAID event.

---

## Kernel trace from the most recent occurrence

```text
2026-09-21T07:13:46+02:00 ZimaOS kernel: WARNING: CPU: 3 PID: 12554 at fs/btrfs/delayed-inode.c:1572 btrfs_insert_delayed_dir_index+0x1e0/0x200
2026-09-21T07:13:46+02:00 ZimaOS kernel: CPU: 3 UID: 999 PID: 12554 Comm: postgres Tainted: P S         O        6.18.9 #4 PREEMPT(voluntary)
2026-09-21T07:13:46+02:00 ZimaOS kernel: btrfs_add_link+0x98/0x460
2026-09-21T07:13:46+02:00 ZimaOS kernel: btrfs_rename+0x627/0xcf0
2026-09-21T07:13:46+02:00 ZimaOS kernel: btrfs_rename2+0x25/0x60
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS: Transaction aborted (error -28)
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info DATA (sub-group id 0) has 17780453376 free, is not full
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info total=1871540387840, used=1853759741952, pinned=0, reserved=208896, may_use=741376, readonly=65536 zone_unusable=0
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info METADATA (sub-group id 0) has 181096448 free, is not full
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info total=6442450944, used=5720768512, pinned=5013504, reserved=1114112, may_use=534511616, readonly=65536 zone_unusable=0
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): global_block_rsv: size 536870912 reserved 536870912
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): trans_block_rsv: size 3538944 reserved 0
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): delayed_block_rsv: size 720896 reserved 720896
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS: error (device md0 state A) in btrfs_add_link:6752: errno=-28 No space left
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state EA): forced readonly
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS: error (device md0 state EA) in btrfs_rename:8624: errno=-28 No space left
```

---

## Previous occurrence showing the same signature

```text
2026-09-14T00:01:03+02:00 ZimaOS kernel: WARNING: CPU: 2 PID: 880737 at fs/btrfs/delayed-inode.c:1572 btrfs_insert_delayed_dir_index+0x1e0/0x200
2026-09-14T00:01:03+02:00 ZimaOS kernel: CPU: 2 UID: 999 PID: 880737 Comm: postgres Tainted: P S         O        6.18.9 #4 PREEMPT(voluntary)
2026-09-14T00:01:03+02:00 ZimaOS kernel: BTRFS: Transaction aborted (error -28)
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info DATA (sub-group id 0) has 18867826688 free, is not full
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info METADATA (sub-group id 0) has 320733184 free, is not full
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info total=6442450944, used=5580685312, pinned=3424256, reserved=147456, may_use=537395200, readonly=65536 zone_unusable=0
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state A): global_block_rsv: size 536870912 reserved 536870912
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state A): trans_block_rsv: size 3407872 reserved 0
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS: error (device md0 state A) in btrfs_add_link:6752: errno=-28 No space left
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state EA): forced readonly
```

---

## Comparison with upstream Btrfs report and merged fix

The upstream report describes filesystems receiving ENOSPC while having more than 100 GB unallocated. Its sample trace contains the same key sequence:

```text
btrfs_insert_delayed_dir_index
btrfs_add_link
Transaction aborted (error -28)
space_info METADATA ... is not full
global_block_rsv ... fully reserved
trans_block_rsv ... reserved 0
forced readonly
```

The reported mechanism is repeated COW of the same B-tree path during `btrfs_search_slot()` under sustained/global memory pressure. This invalidates the assumption that the transaction reservation represents the worst-case requirement, allowing the reserve to become exhausted.

The corresponding fix was subsequently merged upstream:

1. `f9a48549a15aa369d42cebc08a6a72b71a53d547` — **btrfs: inhibit extent buffer writeback to prevent COW amplification**  
   https://github.com/torvalds/linux/commit/f9a48549a15aa369d42cebc08a6a72b71a53d547
2. `67bd829a14b7d71ffea28147b98dadf9ee684568` — **btrfs: replace writeback inhibition xarray with a fixed inline buffer**  
   https://github.com/torvalds/linux/commit/67bd829a14b7d71ffea28147b98dadf9ee684568

The second commit is an important refinement of the first: it removes a reported write-IOPS regression and an unbounded cleanup walk. Therefore, backporting only the original xarray implementation is not recommended; IceWhale should use the final upstream implementation or upgrade to a kernel that contains both commits.

Please confirm whether ZimaOS kernel 6.18.9 contains these changes or an equivalent backport. The runtime behavior strongly indicates that this build remains affected.

## Existing ZimaOS issue search

No exact duplicate was found among open or closed `IceWhaleTech/ZimaOS` issues or pull requests. The closest reports are related but technically different:

- #560: recurring read-only storage and Docker failures on kernel 6.18.9, but on ext4 with EROFS/EIO journal failures;
- #585: Btrfs/md-RAID on 1.7.1, but a boot-time `btrfs_bio_counter_sub()` NULL-dereference panic;
- #601: a duplicate storage-database row that only makes the UI report read-only while the filesystem remains `rw`;
- #326: deleted space held by `icewhale-files-backup`, without a Btrfs transaction abort or forced-readonly transition.

This report therefore describes a separate kernel-level failure and should be tracked as a new issue.

---

## Steps to reproduce

The failure is intermittent and has not been intentionally reproduced because this is a production NAS and the result forces the primary filesystem read-only.

Observed conditions:

1. Start ZimaOS v1.7.1 with Btrfs Main Storage on `md0`.
2. Run a normal multi-container workload for several days.
3. Include concurrent metadata-heavy activity, PostgreSQL checkpoints/renames, and periods of memory/writeback pressure.
4. Eventually a directory-index insertion/rename exhausts `trans_block_rsv`.
5. Btrfs aborts the transaction and forces `md0` read-only.
6. Docker subsequently exits because its data root is no longer writable.

The two observed incidents occurred about one week apart.

---

## Temporary workaround

A controlled reboot remounts Main Storage read-write and allows Docker to start again. This only clears the transient reservation/error state; it does not prevent recurrence.

A direct read-write remount was deliberately not attempted after the aborted transaction.

---

## Additional hardware observation

There is a separate storage-link warning that should be investigated independently. The kernel logged repeated ATA interface errors for one RAID member, a Crucial `CT1000BX500SSD1` exposed as `sdb`:

```text
failed command: READ FPDMA QUEUED
Emask 0x10 (ATA bus error)
status: { DRDY DF ERR }
error: { ICRC ABRT }
SATA link down
hard resetting link
limiting SATA link speed to 3.0 Gbps
```

The drive repeatedly reconnects and mdadm continues to report `[UUUUUUU]`. The `ICRC`, bus-reset, and speed-downgrade pattern suggests a SATA/backplane/power/device-link issue rather than proving Btrfs data-space exhaustion.

This hardware warning should not obscure the separate reproducible Btrfs failure signature: both read-only events show transaction-reserve exhaustion with substantial filesystem capacity still available and match the upstream Btrfs report closely.

---

## Requested action from IceWhale

Please:

1. Confirm whether this is a known issue in ZimaOS v1.7.1 / kernel 6.18.9.
2. Confirm whether the upstream `btrfs_search_slot()` COW-amplification fix is present in this kernel build.
3. Provide a patched kernel or safe ZimaOS update/backport.
4. Advise whether upgrading or rolling back to a specific stable ZimaOS version is the recommended mitigation.
5. Add a prominent ZimaOS alert when a storage filesystem is forced read-only.
6. Consider preventing Docker restart loops after its data root becomes read-only, while preserving diagnostic logs.
7. Advise which diagnostic bundle or full journal should be supplied for further analysis.

Because this is recurring and takes every Docker service offline, a kernel-level resolution or supported rollback path is required; rebooting is not an adequate permanent workaround.


Navigation Menu

[1.7.1][Bug] Btrfs COW amplification exhausts transaction reserve, forces Main Storage read-only and stops Docker #607

Bug report: Btrfs transaction-reserve exhaustion forces Main Storage read-only and stops Docker

Summary

Severity

Environment

Recurrence

Actual result

Expected result

Evidence that the array/filesystem was not actually full

Kernel trace from the most recent occurrence

Previous occurrence showing the same signature

Comparison with upstream Btrfs report and merged fix

Existing ZimaOS issue search

Steps to reproduce

Temporary workaround

Additional hardware observation

Requested action from IceWhale

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[1.7.1][Bug] Btrfs COW amplification exhausts transaction reserve, forces Main Storage read-only and stops Docker #607

Description

Bug report: Btrfs transaction-reserve exhaustion forces Main Storage read-only and stops Docker

Summary

Severity

Environment

Recurrence

Actual result

Expected result

Evidence that the array/filesystem was not actually full

Kernel trace from the most recent occurrence

Previous occurrence showing the same signature

Comparison with upstream Btrfs report and merged fix

Existing ZimaOS issue search

Steps to reproduce

Temporary workaround

Additional hardware observation

Requested action from IceWhale

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions