Switch to streaming decompression

[p2004a]

Right now full replay is decompressed, and then parsed.

It's quite wasteful both from memory usage perspective, but also when somebody just wants to read header and that's it.

This issue is to switch replay parsing to streaming decompression.

[fcaps]

Just tested streaming decompression for headers+script only, to get a feeling how much it would bring.

getHeaders() -> the new function using it.
skipPakets -> what you are prob. using
full parse -> with packets

(Peak Memory) including test-setup etc.

The time is constantly below 1 ms not depending on the file size, as expected.

  ┌──────────────┬───────────────────────┬─────────────────────┐                                                                                                                                                                                                                                                    
  │    Method    │         Time          │     Peak Memory     │                                                                                                                                                                                                                                                      ├──────────────┼───────────────────────┼─────────────────────┤
  │ getHeaders() │ 0.86 ms               │ 0.45 MB             │                                                                                                                                                                                                                                                    
  ├──────────────┼───────────────────────┼─────────────────────┤
  │ skipPackets  │ 17.17 ms (20x slower) │ 7.20 MB (16x more)  │
  ├──────────────┼───────────────────────┼─────────────────────┤
  │ full parse   │ 72.97 ms (85x slower) │ 11.49 MB (25x more) │
  └──────────────┴───────────────────────┴─────────────────────┘

Draft:

# demo-model.ts

   export type HeadersOnly = {
       header: Header;
       info: Info.Info;
       script: string;
   }

#demo-parser.ts

    public static getHeaders(demoFilePath: string): Promise<DemoModel.HeadersOnly> {
        return new Promise((resolve, reject) => {
            const HEADER_SIZE = 352;
            const chunks: Buffer[] = [];
            let totalLength = 0;
            let header: DemoModel.Header | null = null;
            let requiredBytes = HEADER_SIZE;
            let resolved = false;

            const fileStream = createReadStream(demoFilePath);
            const gunzip = createGunzip();

            const cleanup = () => {
                gunzip.removeAllListeners();
                fileStream.removeAllListeners();
                gunzip.destroy();
                fileStream.destroy();
            };

            gunzip.on("data", (chunk: Buffer) => {
                if (resolved) return;

                chunks.push(chunk);
                totalLength += chunk.length;

                if (!header && totalLength >= HEADER_SIZE) {
                    const buffer = Buffer.concat(chunks);
                    const bufferStream = new BufferStream(buffer);
                    header = {
                        magic: bufferStream.readString(16),
                        version: bufferStream.readInt(),
                        headerSize: bufferStream.readInt(),
                        versionString: bufferStream.readString(256, true),
                        gameId: bufferStream.read(16).toString("hex"),
                        startTime: new Date(Number(bufferStream.readBigInt()) * 1000),
                        scriptSize: bufferStream.readInt(),
                        demoStreamSize: bufferStream.readInt(),
                        gameTime: bufferStream.readInt(),
                        wallclockTime: bufferStream.readInt(),
                        numPlayers: bufferStream.readInt(),
                        playerStatSize: bufferStream.readInt(),
                        playerStatElemSize: bufferStream.readInt(),
                        numTeams: bufferStream.readInt(),
                        teamStatSize: bufferStream.readInt(),
                        teamStatElemSize: bufferStream.readInt(),
                        teamStatPeriod: bufferStream.readInt(),
                        winningAllyTeamsSize: bufferStream.readInt(),
                    };
                    requiredBytes = HEADER_SIZE + header.scriptSize;
                }

                if (header && totalLength >= requiredBytes) {
                    resolved = true;
                    cleanup();

                    const buffer = Buffer.concat(chunks);
                    const script = buffer.subarray(HEADER_SIZE, HEADER_SIZE + header.scriptSize);
                    const scriptParser = new ScriptParser({});
                    const scriptInfo = scriptParser.parseScript(script);

                    const meta: DemoModel.Info.Meta = {
                        gameId: header.gameId,
                        engine: header.versionString,
                        game: scriptInfo.hostSettings.gametype,
                        map: scriptInfo.hostSettings.mapname,
                        startTime: header.startTime,
                        durationMs: header.wallclockTime * 1000,
                        fullDurationMs: header.wallclockTime * 1000,
                        winningAllyTeamIds: [],
                        startPosType: parseInt(scriptInfo.hostSettings.startpostype)
                    };

                    resolve({
                        header,
                        info: { meta, ...scriptInfo },
                        script: script.toString(),
                    });
                }
            });

            gunzip.on("error", (err) => {
                if (!resolved) {
                    resolved = true;
                    cleanup();
                    reject(err);
                }
            });

            gunzip.on("end", () => {
                if (!resolved) {
                    resolved = true;
                    cleanup();
                    reject(new Error("Unexpected end of gzip stream before reading complete header and script"));
                }
            });

            fileStream.on("error", (err) => {
                if (!resolved) {
                    resolved = true;
                    cleanup();
                    reject(err);
                }
            });

            fileStream.pipe(gunzip);
        });
    }