Skip to content

Add a script to split build log files into steps (with and without debug output) - #5239

Open
ocaisa wants to merge 2 commits into
easybuilders:developfrom
ocaisa:log_splitter
Open

Add a script to split build log files into steps (with and without debug output)#5239
ocaisa wants to merge 2 commits into
easybuilders:developfrom
ocaisa:log_splitter

Conversation

@ocaisa

@ocaisa ocaisa commented Aug 14, 2026

Copy link
Copy Markdown
Member

This is a helper script to split log files, including the ability to strip out debug output. It should be helpful for locating errors in logs with only the step context, and also for exploring logs after the fact (like verifying configuration step output).

Of course I also expect this to be useful for providing context to AI, and the script itself was generated with the help of ChatGPT and Gemini.

@ocaisa ocaisa added the AI-assisted AI-assisted contributions label Aug 14, 2026
@ocaisa

ocaisa commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

Tweaked to automatically zip the output and not hold all logs in memory. Stress-tested with a debug log for R-bundle-CRAN, which took about 6 minutes to process

easybuild-R-bundle-CRAN-2025.11-20260731.222638_parsed/steps(log_splitter)$ ls -lrh
total 76M
-rw-r--r-- 1 ocaisa ocaisa 3.4K Aug 14 11:18 0017_package_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  929 Aug 14 11:18 0016_permissions_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  34K Aug 14 11:18 0015_module_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  837 Aug 14 11:18 0014_cleanup_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  86K Aug 14 11:18 0013_sanitycheck_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  944 Aug 14 11:18 0012_postproc_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  756 Aug 14 11:18 0011_postiter_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  76M Aug 14 11:18 0010_extensions_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  932 Aug 14 11:18 0009_install_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  861 Aug 14 11:18 0008_test_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  738 Aug 14 11:18 0007_build_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  874 Aug 14 11:18 0006_configure_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa 6.2K Aug 14 11:18 0005_prepare_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  744 Aug 14 11:18 0004_patch_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  746 Aug 14 11:18 0003_extract_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa 1.5K Aug 14 11:18 0002_ready_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  87K Aug 14 11:18 0001_fetch_step.log.bz2
-rw-r--r-- 1 ocaisa ocaisa  77K Aug 14 11:18 0000_init_step.log.bz2

@boegel boegel added this to the 5.x milestone Aug 26, 2026
@boegel

boegel commented Aug 26, 2026

Copy link
Copy Markdown
Member

@ocaisa How difficult would it be to enhance this further to (optionally) get separate logs per extension installation?

That would only make sense when --parallel-extension-install is not used though, of course...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-assisted AI-assisted contributions enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants