An end-to-end engineering showcase of fine-tuning a compact instruction-following language model (Qwen 1.5B) directly on chronological user behavior sequences.
View the Interactive Project Dashboard Here
I built a dedicated, narrative-driven dashboard to explain the mathematical constraints, data pipelines, and architectural decisions behind this project. Please view the dashboard linked above for the most comprehensive overview of this project.
This project investigates whether QLoRA can adapt a small base model to understand chronological user interactions (movie reviews). The pipeline is evaluated on generating a strict, structured JSON churn-risk response.
The Result is Intentionally Honest: While the LLM successfully learned the required formats and syntax structure perfectly, a traditional Logistic Regression baseline outperformed the generative adapter for actual risk scoring (0.71 ROC-AUC vs 0.58 ROC-AUC). This project serves as a reproducible benchmark demonstrating the realities of using LLMs for structured propensity scoring: Format learning does not equal predictive calibration.
Unlike standard LLM fine-tuning tutorials, this project tackles real-world ML engineering challenges under strict hardware constraints (Single 4GB VRAM GPU):
- Strict Temporal Leakage Prevention: Built a custom data pipeline that strictly enforces disjoint user-level splitting. A user exists exclusively in train or test, ensuring the model never "sees the future" of a user it trained on.
- Response-Only Loss Masking: Implemented custom token masking so the PyTorch loss objective ignores the massive prompt history (
-100ignore index) and exclusively optimizes the model on the generated target JSON. - 4-Bit Quantization (QLoRA): Successfully scaled the adapter to rank 128 entirely on consumer hardware.
- Honest Baseline Comparison: Actively built and evaluated a traditional behavioral baseline (using RFM features) to serve as a ground-truth benchmark against the generative model.
Disclaimer: The sample is bounded and public rather than production data. The 30-day inactivity label is a proxy for churn. Privacy, fairness, retention impact, and production readiness are outside this experiment's scope.