Add GASMAN workaround for fewer than 48-bits of addressable memory (riscv) - #6522
Add GASMAN workaround for fewer than 48-bits of addressable memory (riscv)#6522orlitzky wants to merge 2 commits into
Conversation
On riscv, there may not be the full 48 bits of addressable virtual memory that GASMAN is expecting. To facilitate workarounds for this, we add a new ./configure check to detect an SV39 MMU: https://docs.kernel.org/arch/riscv/vm-layout.html The check is skipped if we are cross-compiling, or if the host architecture is anything other than riscv64. The check itself consists of a small C program that queries the Linux kernel for the largest supported virtual address: https://docs.kernel.org/arch/riscv/hwprobe.html Naturally, the check requires Linux to function. If it succeeds, a new preprocessor constant HAVE_SV39_MMU is defined. If it fails, or if it is skipped, we do nothing.
Many 64-bit RISC-V machines have an MMU that supports only 39 bits of addressable virtual memory. On Linux, this is limited to 38 bits: https://docs.kernel.org/arch/riscv/vm-layout.html When GASMAN initially allocates its pool, it suggests to mmap() that the pool should be offset by 16TB to avoid collision with subsequent mallocs. This request is honored, even when only 38 bits are usable, by putting the pool at the end of addressable memory: gap-packages/images#41 This can lead to GASMAN thinking that is has run out of memory on such a system if it later tries to enlarge the pool. The ./configure script is now capable of detecting these RISC-V systems. When one is found, we provide an offset of 96GB instead. This has proved much more reliable in testing.
| /* On Linux (the only place we check for it), this MMU can only | ||
| * address 38 bits of virtual memory. The hint supplied to mmap() | ||
| * therefore needs to be much smaller than the default 16TB. Trial | ||
| * and error shows that 96GB works well enough. */ |
There was a problem hiding this comment.
That's on your 128 GB machine though, isn't it? But will it still work if run in e.g. a 16 GB machine, or a 256 GB machine?
There was a problem hiding this comment.
This is only the offset within the virtual space and should not depend on the amount of physical RAM in the system. (I certainly don't have 16TB installed on the x86_64 box.)
I first tried 128GB (the halfway point in my VM space) but 96GB allowed me to extend the workspace much further. There is a trade-off between how much space to reserve for GASMAN vs. how much to leave for extensions to malloc. By choosing 96GB, I am limiting extensions to 96GB of usable RAM even though the machine has more available. But presently I find that less likely than the 100% chance that the images test suite wants to allocate 128GB.
The fact that this box has so much RAM on an SV39 MMU is an oddity. I don't know what the board designers were thinking, but the firmware is limited to 128GB, and I would guess that future boards (that can handle more physical RAM) will have more capable MMUs.
|
So this doesn't look like something that's going to break anything else. I don't love hard-wiring numbers like this, but if it works fine, it feels fine. There might be a better systematic way of doing it (like trying a steadily decreasing location), but there aren't enough weird architectures that need this to figure out what the best values are. Fundamentally this is just about making sure we have a block of memory, and can also leave space for normal malloc/new. |
|
I could make it more general with |
|
The diagnosis is right, but I think the detection belongs at runtime, not in
So So something like this should work: static const UInt SyPoolHints[] = {
16UL * 1024 * 1024 * 1024 * 1024, // 16 TB
1UL * 1024 * 1024 * 1024 * 1024, // 1 TB
96UL * 1024 * 1024 * 1024, // 96 GB, fits an SV39 address space
};
result = MAP_FAILED;
for (UInt i = 0; i < ARRAY_SIZE(SyPoolHints); i++) {
void *hint = (void *)SyPoolHints[i];
result = mmap(hint, size, PROT_READ|PROT_WRITE, GAP_MMAP_FLAGS, -1, 0);
if (result == hint)
break;
if (result != MAP_FAILED)
munmap(result, size);
result = MAP_FAILED;
}
if (result == MAP_FAILED)
result = mmap(NULL, size, PROT_READ|PROT_WRITE, GAP_MMAP_FLAGS, -1, 0);On SV39 the first two hints exceed the 256 GB user address space and get relocated, so we discard them and land on 96 GB. I believe this what @ChrisJefferson meant be "steadily decreasing location", and it also covers the case the current patch misses: nothing detects the 96 GB hint itself being unavailable. |
That's true in this case but I don't think we can count on it. The most that POSIX guarantees is that
For a given version of linux we can of course read the source for the mmap implementation, but they are free to change it at any time, and we still have three or four flavors of BSD/Solaris to think about. Adding
If implementations return /*
* The following code accomplishes this, but is not portable and
* potentially not safe, since the POSIX standard does not make
* any sufficiently strong promises with regard to the use of
* MAP_FIXED.
*/The benefit of the build-time special case is that we don't have to have any faith in the implementations to do the right thing. If I've made some horrible mistake, the damage is confined to RISC-V users who have an SV39 MMU. That set is small and should remain so, and we can be sure that it has been tested. |
3 participants
Add a
./configurecheck to detect an SV39 MMU on RISC-V hardware, and supply a better hint for GASMAN's initialmmap()when one is found.Tested on riscv64/musl (patch works) and x86_64/glibc (nothing changes).
Closes: gap-packages/images#41