Enable SEEK_HOLE in coreutils 9.0

[dasJ]

Issue description

#143097 will disable SEEK_HOLE in coreutils to fix regressions on ZFS (and potentially other filesystems?).
This issue is to track progress on us re-enabling the feature since it gives us a potential performance boost.
This issue is also to move the discussion around this problem from #141684 since it will probably outlive the staging-next cycle.

Useful links (maintainers feel free to edit and add):

• coreutils bump: [staging] coreutils: 8.32 -> 9.0 #139541
• staging-next PR where the issue surfaced: staging-next 2021-10-15 #141684
• PR that downgrades coreutils for Darwin: Revert "coreutils: 8.32 -> 9.0" for darwin only #142836
• PR that disables SEEK_HOLE in cp: [staging-next] coreutils: Disable SEEK_HOLE due to corruption #143097
• ZFS issue: SEEK_DATA fails randomly openzfs/zfs#11900
• Golang issue: Bootstrapped go binary segfaults immediately in ld.so golang/go#48636
• Coreutils issue: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51433

[mohe2015]

Followup on the peertube issue I and others investigated further:

If you build the esbuild_locked derivation in peertube on top of the buggy staging-next you should be able to reproducibly get a segfaulting esbuild. If you now do an arbitrary derivation-hash changing change on the esbuild_locked derivation you should (with likely high probability) get a working esbuild.
This shows that the issue seems to be rare (but reproducible). Also if you diff these two build results you will see that lots of data was overwritten with zeros which is the cause of the segfault. This symptom is closely matching with the SEEK_HOLE (holes are filled with zeros) issue in coreutils and reverting that commit in coreutils also fixes the issue with esbuild.
The problem was mostly that not many derivations of esbuild on top of the buggy coreutils exhibit the buggy behaviour so it's hard to investigate.

It's interesting that changing the derivation also usually hides the problem and currently not really known why that happens (reproducibly even)

[eggert]

My guess is that coreutils is falling victim to a kernel or file system bug of some sort. (Of course I could be wrong.) See my recent comments in coreutils bug#51433.

[dasJ]

@eggert One thing that I just remembered tonight is the Nix substitution mechanism which may be causing some of the non-ZFS issues. When building something locally (no matter the filesystem), Nix tries to query the builds results from the cache.nixos.org (where the builders are at least partially ZFS-backed iirc).
This means that even when somebody is building locally on ext4, they may get substituions for intermediate build products (like the Go compiler) that were built on ZFS.

The problem is that it's really hard to tell what happened here since paths that are substituted and paths that are built locally cannot really be told from each other and disabling substitution entirely means huge local rebuilds.

[mohe2015]

Cross-Ref #143097 (comment)

TLDR: The peertube issue's root cause is extremely likely zfs on the binary cache as dasJ explained.

[dasJ]

Looks like we can drop the patch for the next coreutils release, the coreutils maintainers deem this as fixed: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51433#58

[dasJ]

Going to close this, as it's enabled now