Skip to content

Repository files navigation

Memory-Centric Clinical LLM — Capacity over TFLOPS

A reproducible, single-machine harness demonstrating that a large clinical LLM (27B, INT4) can be hosted on a consumer CPU using system RAM where a mid-range consumer GPU runs out of VRAM. It self-adapts to the host hardware, runs four experiments, and emits one self-contained report — and the headline result has been independently reproduced on two different consumer machines (an AMD desktop and an Intel workstation laptop), confirming it is not hardware-specific.

Research artifact for an ICCE-style study on memory-centric, privacy-preserving clinical AI at the edge. Not a medical device. Synthetic data only. See Disclaimer.

Headline result — measured on two independent commodity machines: a 27B INT4 model loads entirely into system RAM while the on-board GPU cannot hold it.

Machine A (desktop) Machine B (mobile)
CPU AMD Ryzen 5 5500GT Intel Xeon E-2176M
System RAM 67.2 GB 34.1 GB
27B INT4 weights 16.65 GB 16.65 GB
Peak resident memory 24.98 GB (37.2%) 20.43 GB (59.9%)
GPU GeForce GTX 1060, 6.44 GB VRAM Quadro P1000, 4.29 GB VRAM
GPU verdict OOM OOM

Every number above — plus throughput, energy and LoRA bandwidth — is regenerated by python compare.py, which reads the committed benchmarks/summaries.json and needs no model download.


The idea

The bottleneck for local clinical AI is VRAM, not raw compute. Consumer GPUs cap at 4–16 GB, but a 27B INT4 model's working state is ~17–25 GB. Centralizing the data to use cloud GPUs conflicts with the privacy of clinical records.

This harness reframes the edge node as capacity-enabled (system DDR RAM) rather than compute-limited (GPU VRAM), and measures the trade-off honestly:

Exp Question it answers
E1 Does the INT4 model fit in RAM where it would OOM a consumer GPU?
E2 What inference latency (TTFT) and throughput (tok/s) does a CPU deliver?
E3 What is the energy cost — tokens per joule ("Clinical Intelligence per Watt")?
E4 How much bandwidth does federated LoRA-only sharing save?

The harness is vendor-neutral — pure CPU + system RAM, no GPU and no hardware-accelerator dependency. Running it on a second machine confirms the capacity result generalizes across consumer CPUs; compare.py merges hosts into one table.


Quick start

Requires Python 3.9+, ~3 GB disk (2B model) or ~20 GB (27B), and internet for the first run. No GPU required; no Hugging Face login required.

python -m venv .venv
source .venv/bin/activate                 # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

python src/fetch_models.py --tier edge     # 2B, ~1.7 GB  (or: --tier capacity for 27B)
python src/gen_synthetic_fhir.py           # synthetic clinical prompts (no real PHI)
python run_experiment.py                   # auto-selects the largest model that fits RAM

Output lands in results/<host>-<timestamp>/, including a self-contained final_report.md. Wrappers ./run.sh (Linux/macOS) and run.bat (Windows) are provided.

Useful flags

  • --tier {auto,edge,capacity} — pin the model (use --tier edge on every host for an apples-to-apples throughput comparison; otherwise the tier is auto-selected by available RAM and may differ between machines).
  • --report-only — print the hardware/capability report and exit.
  • --cpu-only — ignore any detected GPU (E1 then reports no GPU comparison).
  • --no-figures — skip figure/table generation (no matplotlib needed).
  • --json — also dump the raw capabilities JSON to stdout.

Reproducing the cross-host comparison

The canonical runs used in the study are committed in benchmarks/summaries.json, so the comparison reproduces with no arguments:

python compare.py
# → results/_comparison/{comparison_table.md, compare_throughput.png, compare_ciw.png}

To add a new machine: run on it (same tier as the others for fair E2/E3), then either drop its final_report.md path on the command line (python compare.py path/to/final_report.md) or append its summary block to benchmarks/summaries.json. final_report.md embeds a machine-readable summary, so a single report file is enough for the merge.

Rows are keyed by host:tier, newest timestamp wins. So a local run on a host whose name and tier match a committed row supersedes that row — expected when re-measuring your own machine, worth knowing if you want the shipped numbers untouched (move or delete results/ first).

Verified environment

The committed runs and the end-to-end check above were performed with Python 3.12 on Linux, llama-cpp-python 0.3.28, psutil 7.2, matplotlib 3.10, huggingface_hub 1.18. Nothing in the harness is version-pinned beyond the floors in requirements.txt; the code parses cleanly under Python 3.9+.


Repository structure

run_experiment.py        master entry: detect → select model → run E1-E4 → report
compare.py               cross-host merge → comparison table + figures
benchmarks/              committed canonical run summaries (the runs used in the study)
requirements.txt         dependencies (llama-cpp-python installed separately, see above)
models.lock              records downloaded model files (repo, size, sha256)
src/
  detect.py              hardware/capability detection (OS, ISA, RAM, GPU, power)
  fetch_models.py        download ungated community GGUF models
  gen_synthetic_fhir.py  synthetic FHIR-shaped data + instruction prompts (non-IID shards)
  backends/llamacpp.py   CPU inference backend (load + generate, timed)
  power.py               RAPL / nvidia-smi / TDP power telemetry
  workload.py            shared inference benchmark (memoized) for E2 & E3
  summarize.py           compact cross-host summary (embedded in final_report.md)
  figures.py             capacity figure + table
  experiments/           E1 capacity · E2 throughput · E3 energy · E4 comms
data/                    synthetic prompts + shards (git-ignored; regenerate with
                         src/gen_synthetic_fhir.py, fixed seed → byte-identical)
models/                  downloaded GGUF weights (git-ignored; see models.lock)
results/                 live outputs (git-ignored; regenerable)

Models & data

  • Models: open Gemma-2 GGUF (Q4_K_M / INT4) from public community repos, used as architecture-faithful stand-ins for MedGemma. No license gate.
  • Data: synthetic FHIR-shaped records (vitals, labs, conditions, meds) and instruction prompts, generated procedurally with a fixed seed — no real patient data, no PhysioNet/MIMIC credentialing. Partitioned into non-IID specialty shards.

Method honesty (read before citing numbers)

This harness is deliberately transparent about what is measured vs modeled:

  • GPU OOM is analytical — derived from weights_bytes > VRAM_bytes, not a forced crash (no CUDA build is attempted). The verdict is definitive nonetheless.
  • Power is estimate-grade — RAPL energy counters are root-only on many hosts; the harness falls back to a labeled TDP estimate (power_method is recorded in output).
  • Throughput (E2) reflects CPU class/generation, not an accelerator — the two test machines differ in age and memory, so their tok/s differ; no hardware accelerator is used or claimed. The capacity result (E1) is the robust, machine-independent finding.
  • Federated training is not run here — E4 (adapter bandwidth) is computed from architecture; full FedLoRA/aggregation are out of scope for this capacity study.

Disclaimer

This is a research artifact, not a medical device, and must not be used for clinical decision-making. All clinical data in this repository is synthetic. Model outputs are not validated for medical accuracy. "MedGemma" is referenced only as a target architecture; the repo ships open Gemma-2 stand-ins.


Citation

If you use this work, please cite it — see CITATION.cff.

License

MIT.

About

Reproducible harness: a 27B INT4 clinical LLM runs in consumer system RAM where a consumer GPU OOMs. CPU-only, synthetic FHIR data, verified on two machines.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages