Force restarting creates maximum GZIP gibberish (Nginx)

[QuiteAFancyEmerald]

Problem:

Occasionally upon doing npm run kill then npm start the application will error. This happens even with GZIP disabled on Nginx. It is possible to be a cache conflict between pm2 and nginx although we cannot find any documentation for it. Originally thought it was a mismatch between GZIP on NGINX and HU LTS. We use GZIP for our DOM masquerading feature where you can disguise pages as .ico files.

Current Solution:

GZIP Gibberish Fix:

• Delete node_modules
• Delete /dist

npm run kill
npm run fresh-start
sudo rm -rf /var/cache/nginx/* 
sudo nginx -t 
sudo systemctl reload nginx

Repeat once or twice to retain normal behavior.

[yetval]

hey! not 100% sure but i think i might know whats going on here

two things that could be causing this:

1. race between the server and a background rebuild - when pm2 starts, InvisiProxyLTS-src-refresh immediately kicks off another build. that build deletes and recreates views/dist while the server is already running. if a request hits in the tiny window between "files copied" and "files gzip-compressed", the server serves plain html as gzip and the decompressor loses it

2. pkill node kills the pm2 daemon too - npm run kill uses pkill node which nukes every node process including pm2 itself. nginx is then left holding stale connections to the old backend, which is porbably why restarting nginx fixes it

quick fixes if this is on the right track:

• build into views/dist-new first, then rename to views/dist - eliminates the race
• swap pkill node for npx pm2 kill; pkill -f "node backend.js" || true

happy to PR if useful!

[QuiteAFancyEmerald]

This is incredibly useful! Feel free to PR

[yetval]

Thank You I have just opened up a PR for this, please LMK your feedback!

[QuiteAFancyEmerald]

PR did not fix the issue sadly and created some issues when it came to production management. Also Scramjet assets were notably missing.

[yetval]

Hey, sorry again about #548 — the rename helped but didn't actually close the window, and the other changes caused the scramjet and pm2 fallout, that's on me.

Took another pass and I'm not 100% sure but I think the build is wiping live views/dist at the start of every rebuild, which means the backend can briefly read a missing loader.html during init and cache a bad fallback for its whole lifetime — would explain why it happens even with nginx GZIP off and why an nginx reload sometimes fixes it. Fix would be one line in run-command.mjs (drop the early rmSync(distFinal), the existing rename at the end still handles the swap).

[QuiteAFancyEmerald]

I see yeah I have done further testing and I am still unsure of the cause if it is even within the application itself or a NGINX bug. More testing required.

[yetval]

Hey, thanks for testing again! I'm fairly confident it's the app and not nginx. The build deletes the live views/dist folder while it runs:

// run-command.mjs
rmSync(distFinal, { force: true, recursive: true });

If the server boots while that folder is gone, it caches the gzipped 404 page and serves it as plain html until a restart, which explains why nginx settings don't matter.

Have you tried removing that line? The rename at the end of the build already swaps the new files in, so it shouldn't be needed.

[yetval]

Ran this against main to confirm: booting the server while the build has views/dist deleted reproduces the exact gibberish, and removing that rmSync line fixes it with identical build output.

One more thing I noticed while measuring: the delete and rename at the end of the build still leave views/dist missing for about 10ms per build. Renaming the old folder aside instead of deleting it first closes that gap too. I can open a PR with both if you want.

[QuiteAFancyEmerald]

Sorry for the late reply but sure!

[yetval]

Sorry for the late reply but sure!

Opened!