🧠 Phi-3 Vision RAG — Multimodal Retrieval-Augmented Generation on Research Papers

This notebook demonstrates a multimodal Retrieval-Augmented Generation (RAG) pipeline that fuses text + image embeddings from academic PDFs with Phi-3 Vision for grounded, context-aware question answering.

💡 While this example uses the “Attention Is All You Need” paper, you can replace it with any PDF — research article, technical report, or documentation — to explore domain-specific RAG behavior.

---

📋 Overview

The system performs the full RAG loop:

1. Parses and extracts text + figures from a PDF.
2. Embeds them using Jina CLIP V1.
3. Stores vectors in a ChromaDB collection for fast similarity retrieval.
4. Retrieves the most relevant chunks for a given query.
5. Uses Microsoft Phi-3 Vision (13B) to synthesize answers grounded in both text and images.

---

🧩 System Components

Module	Purpose
PDF Parsing	Extracts text and figures with fitz (PyMuPDF).
Embedding Model	Uses jinaai/jina-clip-v1 for unified text + image embeddings.
Vector Database	Persists embeddings in ChromaDB for efficient similarity search.
Retriever	Selects top-k relevant chunks for each query.
Phi-3 Vision	Performs multimodal reasoning and answer generation.
Memory Optimization	Implements hybrid GPU/CPU offload and cache management for T4 GPUs.

---

⚙️ Google Colab Ready

This notebook is ready to run on Google Colab — all dependencies are installed directly inside the notebook.

Just open it, run the setup cell, and start experimenting with your own PDFs.

🧩 No external setup required — all installs (torch, transformers, chromadb, pillow, pymupdf, requests, numpy) are handled automatically in the first cell.

---

🧮 Notebook Workflow

1. Parse PDF: Extract text and image data.
2. Embed: Encode with Jina CLIP V1.
3. Store: Save vectors to ChromaDB.
4. Retrieve: Implement similarity-based search.
5. Generate: Load Phi-3 Vision and run multimodal inference.
6. Query: Ask questions about your document.

---

🚀 Example Usage

Text-based query:

answer = generate_answer(
    "What is the purpose of positional encoding in the Transformer model?",
    top_k=2,
    max_tokens=100
)
print(answer)

Image-based query:

answer = generate_answer(
    "Describe what the encoder-decoder attention figure illustrates.",
    top_k=2,
    max_tokens=120
)
print(answer)

To use another PDF, simply change the input file path in the parsing cell — no code modifications needed.

---

⚡ Performance Notes

Optimized for Google Colab GPUs.

GPU	Suitability	Notes
T4 (16 GB)	✅ Functional with hybrid offload	Keep context ≤ 6 K chars, ≤ 3 images, max_tokens ≤ 120
A10 / L4 (24 GB)	🟢 Smooth	Handles longer contexts and multiple figures
A100 / H100 (40–80 GB)	🧠 Best Performance	Enables full-precision inference and complex multimodal queries

For smaller prompts, T4 is sufficient. For large figure-rich queries, use A100.

---

🧹 Memory Tips

• Offloaded weights stored in offload_phi3/ — safe to delete anytime.
• Run torch.cuda.empty_cache() if VRAM fills up.
• Use use_cache=False to avoid DynamicCache warnings.

---

🧠 Customization & Extensions

• Replace the sample PDF with any research paper, design spec, or user manual.
• Tune top_k, max_tokens, and retrieval settings for larger or smaller documents.
• Combine multiple PDFs to create a mini-corpus for domain-specific RAG exploration.

---

📚 Credits

• Paper: Vaswani et al., “Attention Is All You Need” (NeurIPS 2017)
• Models:
  • jinaai/jina-clip-v1
  • microsoft/phi-3-vision-128k-instruct

---

🪪 License

This project is licensed under the MIT License.

---

Made with ❤️ using PyTorch and Transformers.