Skip to content

ScanCode tool not triggered for a coordinate - aircompressor #798

Description

@yashkohli88

The ScanCode tool is failing to trigger for the following coordinate:

git/github/airlift/aircompressor/e139a31be059855a5a70f6994a4ecaa0adbbea7a

Scancode Investigation Summary

Root Cause

  • Large binary test files (mozilla, samba, webster ~50-100MB each in testdata/silesia/)
  • 2 parallel scancode workers each loading files into memory
  • 1000s timeout allows full processing of these huge files

All Test Scenarios

# Scenario Timeout Processes Options Docker RAM Peak Memory Duration Files Completed Result
1 Original failure 1000s 2 - 7.65GB >7.65GB N/A 0 OOM SIGKILL
2 Manual docker exec 1000s 0 - 47GB ~17GB ~22 min 0/3 (all timed out) Success
3 Test: huge timeout 1000000s 1 --quiet 16GB >16GB N/A N/A OOM SIGKILL
4 Test: huge timeout 1000000s 1 --quiet 47GB ~36GB+ N/A N/A OOM SIGKILL
5 Test: --ignore patterns 1000000s 2 --ignore */testdata/* etc. 16GB ~2GB ~2 min N/A (skipped) Success (files excluded)
6 Test: default timeout 120s 2 - 47GB ~2GB ~5 min 0/3 (all timed out) Success
7 Final: original config 1000s 2 - 47GB 27.57GB ~33 min 2/3 (samba timed out) Success

Key Findings

  • Original config (timeout: 1000, processes: 2) needs ~28GB RAM
  • Original Docker limit was 7.65GB -> OOM
  • Manual run with processes: 0 (debug mode) uses less memory (~17GB) but slower
  • Default timeout (120s) skips large test files, only needs ~2GB
  • --ignore patterns can exclude problematic files entirely (~2GB, fastest)
  • Huge timeout (1000000s) forces full binary parsing → 36GB+ → OOM even at 47GB

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Estimate

    None yet

    Start date

    None yet

    TargetDate

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions