Improve available memory estimation

[Pijukatel]

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/utils

Feature

Using this simple resource monitor

import { getMemoryInfo } from '@crawlee/utils';

// Print memory
const memoryInfo = await getMemoryInfo();
const toMB = (bytes) => Math.round(bytes / 1024 / 1024);
console.log('Available memory', {
    totalMB: toMB(memoryInfo.totalBytes),
    freeMB: toMB(memoryInfo.freeBytes),
    usedMB: toMB(memoryInfo.usedBytes),
    mainProcessMB: toMB(memoryInfo.mainProcessBytes),
    childProcessesMB: toMB(memoryInfo.childProcessesBytes),
});

// Allocate ~200 MB
const ALLOCATION_MB = 200;
console.log(`Trying to allocate ${ALLOCATION_MB} MB of memory`);
const ballast = Buffer.alloc(ALLOCATION_MB * 1024 * 1024).fill(1);
console.log(`Allocated ${toMB(ballast.byteLength)} MB of memory`);

// Print memory again
const memoryInfoAfter = await getMemoryInfo();
console.log('Memory after allocation', {
    totalMB: toMB(memoryInfoAfter.totalBytes),
    freeMB: toMB(memoryInfoAfter.freeBytes),
    usedMB: toMB(memoryInfoAfter.usedBytes),
    mainProcessMB: toMB(memoryInfoAfter.mainProcessBytes),
    childProcessesMB: toMB(memoryInfoAfter.childProcessesBytes),
});

And running it with cgroups-imposed constraints, we see it fails to detect the limits. Limits are imposed, which are proven by crash.

1 - Unrestricted run - we see that it consumes around 275 MB (default mem consumption + 200MB ballast )

.../actors/jscrawlee$ systemd-run --user --scope --quiet --no-ask-password -p MemorySwapMax=0 pnpm start

> jscrawlee@0.0.1 start .../actors/jscrawlee
> node src/main.js

Available memory {
  totalMB: 31365,
  freeMB: 17076,
  usedMB: 14289,
  mainProcessMB: 74,
  childProcessesMB: 0
}
Trying to allocate 200 MB of memory
Allocated 200 MB of memory
Memory after allocation {
  totalMB: 31365,
  freeMB: 16863,
  usedMB: 14502,
  mainProcessMB: 275,
  childProcessesMB: 0
}

2 - Restricted run to 190 MB - we see that it is killed when it tries to allocate an additional 200MB ballast

.../actors/jscrawlee$ systemd-run --user --scope --quiet --no-ask-password -p MemoryMax=190M -p MemorySwapMax=0 pnpm start

> jscrawlee@0.0.1 start .../actors/jscrawlee
> node src/main.js

Available memory {
  totalMB: 31365,
  freeMB: 16952,
  usedMB: 14413,
  mainProcessMB: 72,
  childProcessesMB: 0
}
Trying to allocate 200 MB of memory
Killed

3 - Restricted run to 300 MB - we see that it manages to allocate memory for the ballast 200 MB

.../actors/jscrawlee$ systemd-run --user --scope --quiet --no-ask-password -p MemoryMax=300M -p MemorySwapMax=0 pnpm start

> jscrawlee@0.0.1 start .../actors/jscrawlee
> node src/main.js

Available memory {
  totalMB: 31365,
  freeMB: 17041,
  usedMB: 14324,
  mainProcessMB: 71,
  childProcessesMB: 0
}
Trying to allocate 200 MB of memory
Allocated 200 MB of memory
Memory after allocation {
  totalMB: 31365,
  freeMB: 16832,
  usedMB: 14533,
  mainProcessMB: 271,
  childProcessesMB: 0
}

getMemoryInfo fails to detect the memory limits.

Also seems like Docker container started with these arguments will failt to detect correct memory -m 300m --cgroupns=host

Motivation

Improved memory estimation investigated in Python tooling

Ideal solution or implementation, and any additional constraints

Resource reporting is more robust and working correctly in more environments

Alternative solutions or implementations

No response

Other context

No response

[B4nan]

Just wondering, did you test this on v4?

[Pijukatel]

Just wondering, did you test this on v4?

Memory estimation in these cases seems the same on v4; I tried now.

[B4nan]

Yeah, it should, but still better test it there, if we are to fix this, I'd target v4 (and ideally handle it before the stable release).

[Pijukatel]

Btw., these are rather edge cases. Main use cases work well (running on host, running in docker container with the most common setup). So I think we can fix it whenever there is time. Plus, probably we can wait for the Python tooling to solve and properly test it first and port it here afterwards.

[Pijukatel]

Let's wait for the first release of https://github.com/apify/cgroups-sensor as a tested reference for any changes.