Make the renderers plugin directory a package

[Hmkz0x00]

Fixes #1936.

Why pyarrow goes missing

Both specs gather the plugin modules twice, by two different mechanisms:

datas         = ... collect_data_files('volatility3.framework.plugins', include_py_files = True) ...
hiddenimports = ... collect_submodules('volatility3.framework.plugins') ...

collect_data_files walks the filesystem, so it picks up any .py file it
finds. collect_submodules walks pkgutil, and pkgutil will not descend
into a directory with no __init__.py.

volatility3/framework/plugins/renderers has no __init__.py, so the two
disagree:

collect_data_files(..., include_py_files=True)  ->  renderers/parquet_renderer.py   (shipped)
collect_submodules(...)                         ->  []                              (not analysed)

Only hiddenimports feeds pyinstaller's dependency analysis. The renderer is
therefore copied into the binary as an inert data file, its import pyarrow is
never seen, and pyarrow is left out even though the workflow installs it via
pip install -e .[full,cloud,arrow].

This is exactly the "the renderer code does make it into the final binary" part
of the issue.

Why it is only this directory

renderers is the only subdirectory under volatility3/framework/plugins
without an __init__.py; linux, mac, windows, linux/graphics,
linux/malware, linux/tracing, windows/malware and windows/registry all
have one.

The specs also call collect_submodules on volatility3.framework.automagic
and volatility3.framework.symbols. I checked both: automagic is clean, and
the six directories under symbols/windows without an __init__.py
(bigpools, consoles, gui, netscan, services, shimcache) hold nothing
but JSON, which collect_data_files('volatility3.framework') already handles.
So this is the only place the gap bites.

It dates from e7c1126b, which moved the renderers into the new subdirectory.
Before that they sat at volatility3/framework/plugins/parquet_renderer.py,
directly inside a package collect_submodules covers.

What the user sees

The frozen build still advertises both renderers, because the framework
discovers them at runtime with import_files, which uses os.walk and does not
care about __init__.py either:

  -r, --renderer RENDERER
                        Determines how to render the output (quick, none, csv,
                        pretty, json, jsonl, mermaid, arrow, parquet)

Selecting one is an unhandled traceback rather than a clean error:

  File "...\volatility3\framework\plugins\renderers\parquet_renderer.py", line 44, in __init__
    raise RuntimeError("Arrow output format requires the pyarrow package")
RuntimeError: Arrow output format requires the pyarrow package
[PYI-24588:ERROR] Failed to execute script 'vol' due to unhandled exception!

The fix

Add the missing __init__.py, matching the other plugin subpackages. That is
all pkgutil needs. Both specs use the same collect_submodules line, so
neither spec needs editing and volshell.exe is fixed at the same time.

I preferred this over adding hiddenimports = ['pyarrow', 'pyarrow.parquet'] to
the specs, because that names one dependency of one renderer and leaves the
discovery gap in place for the next module added to the directory, and it would
have to be duplicated in both spec files.

Verification

Built vol.spec before and after on Windows with pyarrow installed, and
compared the analysis TOCs:

	pyarrow entries in Analysis-00.toc	vol.exe
before	0	22.8 MB
after	1362	53.7 MB

Running the built exe against win-xp-laptop-2005-06-25.img:

before:  vol.exe -r parquet ... -> RuntimeError, unhandled, exit 1, 0 bytes
after:   vol.exe -r parquet ... -> exit 0, 1601 bytes, PAR1 magic
         vol.exe -r arrow   ... -> exit 0, 1200 bytes

Both outputs read back correctly (23 rows, 2 columns, Variable/Value) with
pq.read_table and pa.ipc.open_stream.

Non-frozen behaviour is unchanged: 197 plugins discovered with and without the
change, zero import failures, identical plugin sets. import_files skips
filenames starting with __, so the new file is not itself imported as a
plugin.

test/renderers/test_parquet_renderers.py passes (4 passed, 4 skipped for the
absent Linux image). ruff format, ruff check and
test/volatility3_code_analysis.py are clean.

test/plugins/windows/windows.py gives an identical 58 failed / 23 passed with
and without the change on my machine, so this is neutral for it. Those failures
are #2013, not this: the _specific_ tests pass a bare path to
--single-location, which the CLI turns into file://///E:\... on Windows.

One thing worth deciding

The exe goes from 22.8 MB to 53.7 MB, because pyarrow brings a lot of .pyd
and Arrow DLLs with it (_dataset_parquet, _acero, _azurefs, _flight
and friends). That is the cost of actually shipping the dependency the build
already installs, but it is a 2.4x jump and you may want it to be a deliberate
choice rather than a side effect of this fix. Measured with pyarrow 25.0.1 on
Python 3.14; CI is on 3.11 so the exact figures will differ.

If the size is unwelcome, the alternative is to drop arrow from the
pip install line in build-pyinstaller.yml and have the frozen build simply
not offer those renderers, but that needs the renderer to fail gracefully
instead of raising an unhandled RuntimeError, which is a separate change.
Happy to do it whichever way you prefer.

Possible follow-up

Nothing in CI would have caught this, and nothing would catch it coming back.
A smoke test in build-pyinstaller.yml between the build and move steps would:

      - name: Check the optional renderers are usable
        run: |
          ./dist/vol.exe -q -r parquet frameworkinfo > frameworkinfo.parquet
          ./dist/vol.exe -q -r arrow frameworkinfo > frameworkinfo.arrow

frameworkinfo needs no memory image, and the renderer is constructed after the
plugin runs, so this reaches the line that raises. --help would not: argparse
exits during parsing, well before renderers[args.renderer]().

I have left it out to keep this to the one file. Happy to add it here or
separately if you want it.

[ikelos]

Thanks very much for the analysis and locating the problem. The issue you raise of vol.exe becoming too large are a concern (and I wonder if there's a way to require the user to have the DLLs installed, and we only carry the python files for it). Either way, this is much appreciated, and once we decide how to deal with the oversized EXE, I'll get this merged. Thanks! 5:)

[Hmkz0x00]

Thanks! I went and measured the size question. The short version: shipping only the python
files and asking the user to supply the DLLs unfortunately cannot work, but there is a
reasonable middle option that gets a good chunk of the size back.

All numbers below are Windows, pyarrow 25.0.1, Python 3.14, one-file build.

Why "python files only" does not work

• pyarrow/__init__.py does from pyarrow.lib import ... at line 71, and lib is the
  compiled extension (lib.cp314-win_amd64.pyd, 3.5 MB). Without the binaries import pyarrow fails on the first line, so the python files on their own are dead weight.
• A frozen one-file build cannot pick up a user-installed pyarrow anyway. sys.path inside
  the running exe is only the _MEI... extraction directory, and PYTHONPATH is ignored. I
  checked with a throwaway one-file exe that imports pyarrow through importlib (so the
  analyser cannot see it): ModuleNotFoundError, both with and without PYTHONPATH pointed
  at a site-packages that does have pyarrow.
• Even with a runtime hook to add a search path, the user's wheel would have to match the
  exact CPython the exe was frozen against (cp311 for CI today), so it would break on every
  Python bump.

Where the size actually comes from

Not from our spec. pyinstaller-hooks-contrib ships hook-pyarrow.py, which is:

hiddenimports = collect_submodules('pyarrow', filter=lambda x: "tests" not in x)
datas = collect_data_files('pyarrow')
binaries = collect_dynamic_libs('pyarrow')

So as soon as anything imports pyarrow the entire package is collected — about 84 MB into
the analysis, including arrow_flight.dll (14.7 MB), arrow_substrait.dll (2.8 MB), the
static .lib import libraries, the C++ headers under pyarrow/include, and pyarrow/tests.

I first tried the obvious thing, adding the unused pyarrow submodules to excludes in the
spec. It barely moves the needle (53.7 -> 52.7 MiB), because the DLLs arrive via
collect_dynamic_libs, not via import analysis.

What does help

Filtering Analysis.binaries and Analysis.datas after the Analysis(...) call, dropping
flight / substrait / dataset / acero and the parts that are not runtime files at all
(.lib, include/, tests/):

build	vol.exe	-r parquet / -r arrow
pyarrow not bundled (effectively today)	22.8 MiB	unhandled RuntimeError
this PR as-is	53.7 MiB	work
this PR + payload filter	42.7 MiB	work, output byte-identical

The filtered build gives byte-for-byte identical -r parquet and -r arrow output to the
unfiltered one (frameworkinfo, 456 x 3, valid PAR1 and a readable IPC stream).

About 42 MiB looks like the floor if pyarrow ships at all: arrow.dll alone is 22 MB, plus
arrow_compute.dll at 9.4 MB and parquet.dll at 6.9 MB.

Options

1. Merge as-is: 53.7 MiB.
2. Merge, then I add the payload filter to vol.spec and volshell.spec: 42.7 MiB with the
   renderers fully working. Cost is roughly 25 lines in each spec plus a list of DLL names
   that could go stale with a future pyarrow — which is exactly why I would pair it with the
   smoke test I mentioned, so a wrong exclusion fails the build instead of a user's run.
3. Drop arrow from the pip install line in build-pyinstaller.yml: back to 22.8 MiB with
   no parquet/arrow in the frozen builds. I checked the build is happy without pyarrow, but
   the renderer would need to fail cleanly instead of the current unhandled traceback.

The __init__.py in this PR is worth having under any of the three. Under (3) it matters
less visibly: it makes the two collectors agree so the module's dependencies are actually
analysed, rather than the file being copied in without anything ever looking at it — which
is what made this so hard to spot in the first place.

Happy to push (2) or (3) onto this branch, or do it as a follow-up. Just tell me which you
prefer.

[ikelos]

I agree that we should have the init.py for completeness, but I think arrow may have to be a feature we don't support in the exe simply because it adds too much weight for not enough value. I was hoping there was a way that the DLLs could be found on the system automatically rather than us shipping them, but if not I'd almost sooner keep it as is (which I believe should simply not offer arrow support because the import should fail).

I've changed it slightly so that if the arrow libraries can't be found the renderers don't display (there's still a debug message to say they couldn't found, but otherwise a normal user will have no indication). If there's demand we could offer a complete version (the 53Mb version), but otherwise I think the most common use case will be the slim version.

I'll merge this once the tests have completed successfully. Thanks for your contribution!