Skip to content

Latest commit

 

History

History
286 lines (214 loc) · 13.2 KB

README.md

File metadata and controls

286 lines (214 loc) · 13.2 KB

Azure Companion for Retrieval-Augmented Generation

I created this repository as an independently authored set of companion notebooks for learners taking DeepLearning.AI's Retrieval Augmented Generation (RAG) course. It follows the course's broad five-module learning progression while exploring the concepts through Azure AI Search, Azure OpenAI, Azure AI Foundry, and credential-free local implementations.

This project is not affiliated with, endorsed by, or maintained by DeepLearning.AI. It does not reproduce or replace the official lessons, videos, notebooks, or assessments. The labs and assignments here are original materials designed to give Azure users another way to practise the underlying concepts.

20 notebooks — 14 labs, 5 graded assignments, 1 setup notebook.

Every notebook runs with or without an Azure subscription. Without credentials, the labs use local implementations of the same core techniques, so you can complete all companion labs and assignments without cloud charges.


How to use this companion

The official course teaches the concepts and provides its own examples and assessments. Use this repository alongside it when you want to implement the same categories of RAG capability with services commonly used in Azure-based organisations.

Official course area Emphasis in this repository
RAG overview A small, inspectable pipeline with citations and failure diagnosis
Information retrieval BM25, semantic search, hybrid retrieval, and metrics
Vector databases HNSW concepts and Azure AI Search vector indexes
LLMs and generation Azure OpenAI, grounded prompting, and agentic RAG
Production RAG Azure AI evaluation, observability, cost, security, and scaling

A practical rhythm is to study an official module, run the corresponding notebooks here, and compare the design choices. Topic alignment does not imply exercise-for-exercise equivalence.


Quick start

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
jupyter lab setup/00_environment_setup.ipynb

That notebook checks your environment, downloads the corpus, and tells you what will run in Azure mode versus local mode. Start there.

To connect real Azure resources (optional):

./setup/provision.sh                      # API keys
./setup/provision.sh --keyless            # Entra ID + RBAC, no keys on disk
./setup/provision.sh --subscription <id>  # pick the subscription explicitly

Requires the Azure CLI and az login. It creates a resource group, an Azure AI Search service, an Azure OpenAI account with two model deployments, and writes your .env. It prints which subscription it is using before it creates anything.

Already have Azure resources? Copy .env.example to .env and fill in what you have — one service at a time is fine, and anything you leave as a <placeholder> falls back to local mode. When you're done:

./setup/provision.sh --destroy

Companion structure

Module Notebooks Topics
0 · Setup 00_environment_setup Environment check, corpus download, cost expectations
1 · RAG Overview lab1_1 · lab1_2 · assignment The three components; diagnosing which stage failed; first pipeline on Azure AI Search
2 · Information Retrieval lab2_1 · lab2_2 · lab2_3 · assignment Inverted indexes, TF‑IDF, BM25 from scratch; embeddings; hybrid search + RRF; recall/precision/MRR/nDCG
3 · Vector Databases lab3_1 · lab3_2 · lab3_3 · assignment HNSW built from scratch; quantization & truncation; Azure vector indexes; chunking strategy; reranking & late interaction
4 · LLMs and Generation lab4_1 · lab4_2 · lab4_3 · assignment Temperature/top‑k/top‑p; token budgets; grounded prompting; hallucination detection; tool-calling agents; Foundry Agents
5 · Production lab5_1 · lab5_2 · lab5_3 · assignment Azure AI Evaluation SDK; LLM-as-judge; tracing & cost models; caching; prompt injection; access control; scaling

Each lab ends with a Key points summary. Each assignment is auto-graded with a pass mark of 80% and has worked solutions at the bottom.


The two modes

Azure mode Local mode
Retrieval Azure AI Search (BM25, HNSW, semantic ranker) in-memory BM25 + exact KNN
Embeddings text-embedding-3-* TF‑IDF + truncated SVD (LSA)
Generation Azure OpenAI chat deployment (gpt-5-mini by default) extractive sentence selector
Cost pay per token + hourly service zero

Local mode is not a stub — it's a working implementation of the same algorithms. What you lose is the managed service itself: real ANN indexes at scale, the trained cross-encoder reranker, and fluent generation.

Suggested path: Modules 1–2 in local mode (they're about algorithms), then provision Azure for Modules 3–5 where the managed service is the subject.

Set RAG_COURSE_FORCE_LOCAL=1 to disable Azure calls even when credentials exist.


The corpus

25 documents of real Azure AI technical documentation, downloaded from the public MicrosoftDocs/azure-ai-docs repository and cached in data/corpus.json, which is tracked in git.

The download is pinned to a commit, not to main. That repository changes daily, and the corpus is an input to nearly every number these labs print — so a floating corpus would mean your BM25 scores, your chunk counts and your recall figures quietly disagree with the text explaining them. To move to a newer snapshot, bump CORPUS_COMMIT in ragcourse/corpus.py, run python -m ragcourse.corpus --refresh, and run the tests to see what moved.

The Azure AI Search articles are fetched as real Markdown. The Azure OpenAI articles in that repository are INCLUDE-transclusion stubs of a few hundred bytes, so those five documents ship bundled with this repository. If the download fails entirely, a complete offline corpus is used and nothing breaks.

Evaluation uses 15 queries with heuristic relevance labels — a chunk is relevant if it contains the query's required terms. These are "silver" labels, and the notebooks are explicit about what that costs you: Module 2 shows how to sanity-check them, Module 3 shows an experiment they distort, and Module 5 replaces them with an LLM judge and compares the two.


Repository layout

ragcourse/              shared package — import this from any notebook
  config.py             environment detection and capability flags
  corpus.py             corpus download, caching, chunking, gold queries
  fallback_corpus.py    bundled offline documents
  embeddings.py         AzureOpenAIEmbedder + LocalLSAEmbedder (disk-cached)
  llm.py                AzureOpenAIChat + LocalExtractiveLLM, cost tracking
  search_local.py       BM25Index, VectorIndex, HybridIndex, RRF
  azure_search.py       index creation, upload, and querying
  rag.py                RAGPipeline, prompt assembly, RAGAnswer
  metrics.py            recall/precision/MRR/nDCG/MAP + groundedness proxies
  grading.py            the assignment auto-grader

setup/
  00_environment_setup.ipynb
  provision.sh          az CLI provisioning + teardown
  build_notebooks.py    headless execution check for every notebook

tests/                  offline test suite (see Verification)

data/
  corpus.json           tracked: pins the exact corpus every learner retrieves over
  embedding_cache/      generated, untracked
  eval_data.jsonl       generated by Module 5 Lab 1, untracked

Verification

Every claim in this section is checked automatically on each push, on Python 3.11 and 3.12, against both the declared minimum dependency versions and the pinned set in requirements-lock.txt. Nothing in CI touches Azure.

pip install -r requirements-dev.txt

pytest                                    # unit tests + assignment solutions
pytest -m "not slow"                      # skip the notebook-executing cases
python setup/build_notebooks.py --execute # all 20 notebooks, offline

All 20 notebooks execute end to end in local mode with no errors — about 80 seconds for the full sweep.

Every assignment's worked solutions score full marks against its own grader. This is a real test, not a note: tests/test_assignment_solutions.py lifts the reference implementations out of each notebook's Solutions section, splices them into the exercise cells, runs the notebook, and fails if a single check is left unpassed.

Assignment Worked solutions score
Module 1 20.5 / 20.5
Module 2 23 / 23
Module 3 20.5 / 20.5
Module 4 22.5 / 22.5
Module 5 22.5 / 22.5

These are local-mode totals. They are also fixed totals: a second test executes each assignment with every answer left untouched and fails unless it is scored out of exactly the same denominator. That matters because the pass mark is a percentage — if skipping an exercise also removed its points from the total, 80% would mean something different for every attempt.

Notebooks are shipped without saved outputs — you run the cells.

What CI cannot check

Nothing above touches Azure, so none of the Azure branches — index creation, upload, the four query modes, the embedding and chat clients — is covered by a test that runs automatically. setup/azure_smoke_test.py covers them against real resources:

python setup/azure_smoke_test.py --yes

It costs a few cents, builds a scratch index named <your-index>-smoke so it cannot disturb the one the notebooks build, and deletes it afterwards. Worth running after provisioning and after any SDK version bump.


SDK versions

Verified against azure-search-documents 12.0.0 (April 2026) and 11.6.0. The notebooks deliberately use only the API surface common to both, so either line works. Note that 12.0.0 removed serialize()/deserialize() in favour of as_dict(), and several compression model names moved — Module 3 shows how to inspect what your installed version actually exposes.

Also verified: azure-ai-projects 2.4.0, azure-ai-agents 1.1.0, azure-ai-evaluation 1.18.3, openai 2.52.0.


A note on the results

Several labs produce results that contradict common advice:

  • Reranking makes things worse on this corpus (Module 3, Lab 3) — and the lab explains why rather than tuning until it looks good.
  • Standard retrieval metrics cannot choose a chunk size, because the labels move with the chunking. Module 3 builds a boundary-independent measure instead.
  • A semantic cache scores an antonym higher than a true paraphrase (Module 5, Lab 2), so no threshold separates them safely.

These are kept as they are. The habit these companion materials try to build is measuring techniques on your own data rather than adopting them because they're well-regarded.


Cost

If you run all companion notebooks in Azure mode:

Item Approximate cost
Azure AI Search, Basic tier ~$0.10/hour, billed while the service exists
Embedding the corpus (~350 chunks, once) under $0.01
All chat generation across the labs a few cents with gpt-5-mini
Semantic ranker free tier covers 1,000 queries/month

The search service is the only meaningful cost and it accrues hourly whether or not you use it. Embeddings are cached to data/embedding_cache/ keyed by a hash of the text, so re-running a notebook doesn't re-bill you.

Run ./setup/provision.sh --destroy when you finish.


License and attribution

The code, labs, assignments, and bundled fallback prose were written for this repository and are released under the MIT License. The high-level module organisation was informed by DeepLearning.AI's public RAG course syllabus; no official course material is included. When online, the corpus loader downloads and transforms selected articles from MicrosoftDocs/azure-ai-docs, which are available under CC BY 4.0. See Third-Party Notices for full attribution and the non-affiliation statement.