A reproducible, single-machine harness demonstrating that a large clinical LLM (27B, INT4) can be hosted on a consumer CPU using system RAM where a mid-range consumer GPU runs out of VRAM. It self-adapts to the host hardware, runs four experiments, and emits one self-contained report — and the headline result has been independently reproduced on two different consumer machines (an AMD desktop and an Intel workstation laptop), confirming it is not hardware-specific.
Research artifact for an ICCE-style study on memory-centric, privacy-preserving clinical AI at the edge. Not a medical device. Synthetic data only. See Disclaimer.
Headline result — measured on two independent commodity machines: a 27B INT4 model loads entirely into system RAM while the on-board GPU cannot hold it.
| Machine A (desktop) | Machine B (mobile) | |
|---|---|---|
| CPU | AMD Ryzen 5 5500GT | Intel Xeon E-2176M |
| System RAM | 67.2 GB | 34.1 GB |
| 27B INT4 weights | 16.65 GB | 16.65 GB |
| Peak resident memory | 24.98 GB (37.2%) | 20.43 GB (59.9%) |
| GPU | GeForce GTX 1060, 6.44 GB VRAM | Quadro P1000, 4.29 GB VRAM |
| GPU verdict | OOM | OOM |
Every number above — plus throughput, energy and LoRA bandwidth — is regenerated
by python compare.py, which reads the committed
benchmarks/summaries.json and needs no model
download.
The bottleneck for local clinical AI is VRAM, not raw compute. Consumer GPUs cap at 4–16 GB, but a 27B INT4 model's working state is ~17–25 GB. Centralizing the data to use cloud GPUs conflicts with the privacy of clinical records.
This harness reframes the edge node as capacity-enabled (system DDR RAM) rather than compute-limited (GPU VRAM), and measures the trade-off honestly:
| Exp | Question it answers |
|---|---|
| E1 | Does the INT4 model fit in RAM where it would OOM a consumer GPU? |
| E2 | What inference latency (TTFT) and throughput (tok/s) does a CPU deliver? |
| E3 | What is the energy cost — tokens per joule ("Clinical Intelligence per Watt")? |
| E4 | How much bandwidth does federated LoRA-only sharing save? |
The harness is vendor-neutral — pure CPU + system RAM, no GPU and no
hardware-accelerator dependency. Running it on a second machine confirms the
capacity result generalizes across consumer CPUs; compare.py merges hosts into
one table.
Requires Python 3.9+, ~3 GB disk (2B model) or ~20 GB (27B), and internet for the first run. No GPU required; no Hugging Face login required.
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
python src/fetch_models.py --tier edge # 2B, ~1.7 GB (or: --tier capacity for 27B)
python src/gen_synthetic_fhir.py # synthetic clinical prompts (no real PHI)
python run_experiment.py # auto-selects the largest model that fits RAMOutput lands in results/<host>-<timestamp>/, including a self-contained
final_report.md. Wrappers ./run.sh (Linux/macOS) and run.bat (Windows) are
provided.
--tier {auto,edge,capacity}— pin the model (use--tier edgeon every host for an apples-to-apples throughput comparison; otherwise the tier is auto-selected by available RAM and may differ between machines).--report-only— print the hardware/capability report and exit.--cpu-only— ignore any detected GPU (E1 then reports no GPU comparison).--no-figures— skip figure/table generation (no matplotlib needed).--json— also dump the raw capabilities JSON to stdout.
The canonical runs used in the study are committed in
benchmarks/summaries.json, so the comparison
reproduces with no arguments:
python compare.py
# → results/_comparison/{comparison_table.md, compare_throughput.png, compare_ciw.png}To add a new machine: run on it (same tier as the others for fair E2/E3), then
either drop its final_report.md path on the command line
(python compare.py path/to/final_report.md) or append its summary block to
benchmarks/summaries.json. final_report.md embeds a machine-readable summary,
so a single report file is enough for the merge.
Rows are keyed by host:tier, newest timestamp wins. So a local run on a host
whose name and tier match a committed row supersedes that row — expected
when re-measuring your own machine, worth knowing if you want the shipped
numbers untouched (move or delete results/ first).
The committed runs and the end-to-end check above were performed with Python
3.12 on Linux, llama-cpp-python 0.3.28, psutil 7.2, matplotlib 3.10,
huggingface_hub 1.18. Nothing in the harness is version-pinned beyond the
floors in requirements.txt; the code parses cleanly under Python 3.9+.
run_experiment.py master entry: detect → select model → run E1-E4 → report
compare.py cross-host merge → comparison table + figures
benchmarks/ committed canonical run summaries (the runs used in the study)
requirements.txt dependencies (llama-cpp-python installed separately, see above)
models.lock records downloaded model files (repo, size, sha256)
src/
detect.py hardware/capability detection (OS, ISA, RAM, GPU, power)
fetch_models.py download ungated community GGUF models
gen_synthetic_fhir.py synthetic FHIR-shaped data + instruction prompts (non-IID shards)
backends/llamacpp.py CPU inference backend (load + generate, timed)
power.py RAPL / nvidia-smi / TDP power telemetry
workload.py shared inference benchmark (memoized) for E2 & E3
summarize.py compact cross-host summary (embedded in final_report.md)
figures.py capacity figure + table
experiments/ E1 capacity · E2 throughput · E3 energy · E4 comms
data/ synthetic prompts + shards (git-ignored; regenerate with
src/gen_synthetic_fhir.py, fixed seed → byte-identical)
models/ downloaded GGUF weights (git-ignored; see models.lock)
results/ live outputs (git-ignored; regenerable)
- Models: open Gemma-2 GGUF (Q4_K_M / INT4) from public community repos, used as architecture-faithful stand-ins for MedGemma. No license gate.
- Data: synthetic FHIR-shaped records (vitals, labs, conditions, meds) and instruction prompts, generated procedurally with a fixed seed — no real patient data, no PhysioNet/MIMIC credentialing. Partitioned into non-IID specialty shards.
This harness is deliberately transparent about what is measured vs modeled:
- GPU OOM is analytical — derived from
weights_bytes > VRAM_bytes, not a forced crash (no CUDA build is attempted). The verdict is definitive nonetheless. - Power is estimate-grade — RAPL energy counters are root-only on many hosts; the
harness falls back to a labeled TDP estimate (
power_methodis recorded in output). - Throughput (E2) reflects CPU class/generation, not an accelerator — the two test machines differ in age and memory, so their tok/s differ; no hardware accelerator is used or claimed. The capacity result (E1) is the robust, machine-independent finding.
- Federated training is not run here — E4 (adapter bandwidth) is computed from architecture; full FedLoRA/aggregation are out of scope for this capacity study.
This is a research artifact, not a medical device, and must not be used for clinical decision-making. All clinical data in this repository is synthetic. Model outputs are not validated for medical accuracy. "MedGemma" is referenced only as a target architecture; the repo ships open Gemma-2 stand-ins.
If you use this work, please cite it — see CITATION.cff.
MIT.