rh-ai-quickstart · slotkasten · Apr 6, 2026 · Sep 1, 2026 · Sep 1, 2026 · Sep 1, 2026
diff --git a/README.md b/README.md
@@ -1,5 +1,5 @@
 <!-- omit from toc -->
-# Centralize company knowledge with an Enterprise RAG Chatbot
+# Centralize company knowledge with an Enterprise RAG Chatbot and Protect it with Veeam Kasten
 
 [![GitHub release](https://img.shields.io/github/v/release/rh-ai-quickstart/RAG)](https://github.com/rh-ai-quickstart/RAG/releases)
 [![Quay.io](https://img.shields.io/badge/quay.io-llamastack--dist--ui-blue)](https://quay.io/repository/rh-ai-quickstart/llamastack-dist-ui)
@@ -9,6 +9,7 @@ Use retrieval-augmented generation (RAG) to enhance large language models with s
 <!-- omit from toc -->
 ## Table of Contents
 - [Detailed description](#detailed-description)
+  - [Data protection and resilience with Veeam Kasten](#data-protection-and-resilience-with-veeam-kasten)
   - [Architecture diagrams](#architecture-diagrams)
 - [Requirements](#requirements)
   - [Minimum hardware requirements](#minimum-hardware-requirements)
@@ -18,6 +19,7 @@ Use retrieval-augmented generation (RAG) to enhance large language models with s
   - [Prerequisites](#prerequisites)
   - [Supported Models](#supported-models)
   - [Installation Steps](#installation-steps)
+  - [Protect the Deployment with Veeam Kasten](#protect-the-deployment-with-veeam-kasten)
   - [Local Deployment](#local-deployment)
 - [Tags](#tags)
 
@@ -37,11 +39,62 @@ This QuickStart allows users to explore the capabilities of RAG by:
 - Switching between simple and agent based RAG
 
 
+### Data protection and resilience with Veeam Kasten
+
+A production RAG chatbot is not a stateless application. The intelligence that makes FantaCo's assistant useful lives in state that is scattered across the namespace: vector embeddings and their relational metadata in PostgreSQL + PGVector, ingested source documents in S3/MinIO object storage, LlamaStack configuration and session data, Kubeflow pipeline definitions and run history, model-serving configuration, and the ConfigMaps, Secrets, and Routes that wire it all together. Lose any one of those pieces and the chatbot does not simply stop — it keeps answering, but with stale, incomplete, or wrong context. Rebuilding it from scratch means re-running ingestion and re-embedding the entire corpus, consuming hours of GPU/CPU time while the business waits.
+
+**[Veeam Kasten](https://www.veeam.com/products/cloud/kubernetes-data-protection.html) is a Kubernetes-native data management platform** built specifically for containerized applications. Rather than treating a cluster as a set of volumes to be snapshotted, Kasten understands the *application*: it discovers the workloads, custom resources, configuration, and persistent data that constitute a running app, then captures, moves, and restores them as a coherent unit. This QuickStart ships a complete Kasten configuration so you can experience that model against a real AI workload — see [Protect the Deployment with Veeam Kasten](#protect-the-deployment-with-veeam-kasten).
+
+![How Veeam Kasten protects the RAG AI application on OpenShift](docs/images/kasten-rag-protection.png)
+
+*Kasten K10 runs in its own `kasten-io` namespace, alongside — not inside — the RAG application: the `rag-daily-backup` policy selects the RAG namespace, snapshots every PVC through CSI, runs the Kanister blueprint against pgvector, and exports the resulting restore point to the location profile you define. Editable source: [`kasten-rag-protection.svg`](docs/images/kasten-rag-protection.svg).*
+
+#### One application, one restore point
+
+Kasten captures the entire OpenShift AI RAG application — the vector database, the relational metadata that indexes it, the object-store document corpus, and every Kubernetes object that defines the deployment — into a **single, application-consistent restore point**.
+
+| RAG component | State protected by Kasten |
+|---------------|---------------------------|
+| PostgreSQL + PGVector | Vector embeddings, collections, and relational schema and metadata |
+| S3 / MinIO document store | Source documents that feed the ingestion pipeline |
+| LlamaStack | Configuration, registered models and shields, session state |
+| Kubeflow / Data Science Pipelines | Pipeline definitions, run history, and artifacts |
+| Model serving (vLLM / KServe) | InferenceService and serving-runtime definitions |
+| Application plumbing | Deployments, StatefulSets, Services, Routes, ConfigMaps, Secrets, RBAC, CRs |
+
+The distinction matters. Backing up the vector database on its own leaves you with embeddings that no longer match the documents they were derived from, or a database that no application knows how to reach. A Kasten restore point is internally consistent across *all* of it, so a recovery returns a working chatbot rather than a pile of correlated-but-unaligned data that an engineer has to reassemble by hand.
+
+#### Orchestration, not just snapshots
+
+Stateful services need more than a volume copy. Kasten's orchestration engine — built on the open-source [Kanister](https://kanister.io/) framework — lets each workload define exactly *how* it should be quiesced, captured, and brought back:
+
+- **Application-aware capture.** The [`pgvector-logical-backup` blueprint](deploy/helm/kasten/templates/pgvector-blueprint.yaml) included here runs a `pg_dump` against the live pgvector instance and streams it directly to your backup target. Because the result is a logical dump rather than a block-level snapshot, it restores cleanly across different storage classes, clusters, and even PostgreSQL versions — the foundation for migration and dev/test cloning, not only recovery.
+- **Automatic discovery.** Kasten finds the workloads that need special handling through a single annotation (`kanister.kasten.io/blueprint`), applied to the pgvector StatefulSet automatically at install time. Adding protection for another stateful service is a blueprint and an annotation — not a new backup pipeline.
+- **Sequenced, dependency-aware recovery.** On restore, Kasten replays the operation in the correct order: recreate namespace resources, restore persistent volumes, then run the blueprint's logical restore phase, so pods come back attached to the right data with no manual sequencing by an operator at 2 a.m.
+- **Policy-driven and declarative.** The [backup policy](deploy/helm/kasten/templates/rag-policy.yaml) shipped here protects everything in the RAG namespace on a schedule with tiered retention (7 daily / 4 weekly / 12 monthly / 5 yearly), exported off-cluster to a location profile. It is a Kubernetes custom resource delivered by Helm, so protection is versioned and deployed alongside the application itself and fits naturally into a GitOps workflow.
+- **Programmable, event-driven backups.** Kasten's API can be invoked from anywhere in your automation. This QuickStart wires it into the [data ingestion pipeline](notebooks/data-ingestion-pipeline.ipynb): a `trigger_kasten_backup` step calls the K10 `RunAction` API and blocks until the backup completes *before* any new documents mutate the vector database. Every ingestion run is therefore preceded by a known-good rollback point — so a bad document batch, a corrupted embedding run, or a poisoned source corpus is an undo operation rather than an incident.
+
+#### Lower RTO, lower operational overhead
+
+Without a single restore point, recovering this application is a multi-team, multi-hour project: re-provision the namespace, restore or rebuild PostgreSQL, re-seed object storage, re-run ingestion and re-embed the corpus, reconcile secrets and routes, then validate that retrieval quality actually came back. Every step is manual, ordered, and easy to get wrong under pressure.
+
+With Kasten it is one restore action against one restore point:
+
+- **RTO measured in minutes, not hours or days.** Recovery becomes a single orchestrated operation instead of a sequence of hand-run procedures — and because the embeddings are restored rather than regenerated, you skip the re-ingestion and re-embedding compute entirely.
+- **No bespoke backup tooling to maintain.** One policy replaces per-component scripts, cron jobs, and tribal knowledge. Protection is declared once, in the same Helm workflow that deploys the application.
+- **Predictable, testable recovery.** Restores can be rehearsed into a separate namespace or cluster on demand, turning DR from an assumption into something you have actually verified.
+- **Reuse beyond recovery.** The same restore point powers cluster migration, environment cloning for evaluation and testing, and ransomware resilience through immutable, off-cluster backup copies.
+
+For an AI platform team, the practical result is that the RAG application's most valuable and most expensive-to-rebuild asset — its knowledge — is protected with the same rigor, and the same automation, as the code that serves it.
+
+
 ### Architecture diagrams
 ![RAG System Architecture](docs/images/rag-architecture.png)
 
 *This diagram illustrates both the ingestion pipeline for document processing and the RAG pipeline for query handling. For more details click [here](docs/rag-reference-architecture.md).*
 
+For how this deployment is backed up, restored, and moved between clusters, see the data protection architecture in [Data protection and resilience with Veeam Kasten](#data-protection-and-resilience-with-veeam-kasten).
+
 | Layer/Component | Technology | Purpose/Description |
 |-----------------|------------|---------------------|
 | **Orchestration** | OpenShift AI | Container orchestration and GPU acceleration |
@@ -55,6 +108,7 @@ This QuickStart allows users to explore the capabilities of RAG by:
 | **Retrieval** | Vector Search | Retrieves relevant documents based on query similarity |
 | **Data Ingestion** | Kubeflow Pipelines | Multi-modal data ingestion with preprocessing pipelines for cleaning, chunking, and embedding generation |
 | **Storage** | S3 Bucket | Document source for enterprise content |
+| **Data Protection** | Veeam Kasten | Application-consistent backup, restore, and mobility for the entire RAG namespace, including the pgvector database |
 
 
 ## Requirements 
@@ -71,6 +125,7 @@ This QuickStart allows users to explore the capabilities of RAG by:
 - OpenShift Cluster 4.18+
 - OpenShift AI
 - Helm CLI - helm
+- Veeam Kasten (optional, for data protection) - a CSI driver with `VolumeSnapshot` support and an off-cluster object store for backup exports
 
 ### Required user permissions 
 - Regular user permission for default deployment
@@ -308,12 +363,78 @@ oc get routes -n llama-stack-rag
 
 For detailed post-installation verification, configuration options, and usage instructions, see the [complete OpenShift deployment guide](docs/openshift_setup_guide.md).
 
+### Protect the Deployment with Veeam Kasten
+
+Once the RAG application is running, deploy Veeam Kasten to protect it. See [Data protection and resilience with Veeam Kasten](#data-protection-and-resilience-with-veeam-kasten) for why this matters and for an architecture diagram of how the pieces fit together. The chart in [`deploy/helm/kasten`](deploy/helm/kasten) installs Kasten K10, the pgvector Kanister blueprint, and a daily backup policy covering the whole RAG namespace.
+
+1. **Install Kasten K10 and the RAG data protection configuration**
+
+From the `deploy/helm` directory:
+
+```bash
+make install-kasten NAMESPACE=llama-stack-rag
+```
+
+This installs Kasten K10 into the `kasten-io` namespace with OpenShift token authentication, deploys the `pgvector-logical-backup` Kanister blueprint, applies the `kanister.kasten.io/blueprint` annotation to the pgvector StatefulSet, and creates the `rag-daily-backup` policy.
+
+2. **Create a Location Profile**
+
+Open the Kasten dashboard and log in with your OpenShift token:
+
+```bash
+oc get route k10 -n kasten-io
+```
+
+Under **Settings > Locations**, create a location profile pointing at your off-cluster backup storage (S3, Azure Blob, GCS, NFS, or similar).
+
+3. **Bind the profile to the backup policy**
+
+```bash
+make kasten-set-profile NAMESPACE=llama-stack-rag LOCATION_PROFILE=<profile-name>
+```
+
+4. **Verify protection**
+
+```bash
+make kasten-status
+```
+
+This shows the K10 pods, the dashboard route, the backup policies, and the registered Kanister blueprints. Trigger the policy from the dashboard to create your first restore point and confirm the pgvector blueprint phase runs.
+
+**Optional: back up before every ingestion run.** The [data ingestion pipeline](notebooks/data-ingestion-pipeline.ipynb) can trigger a Kasten backup before it writes to the vector database. Set the following environment variables before running the notebook; if they are unset, the step is a no-op and the pipeline behaves as before.
+
+| Variable | Description |
+|----------|-------------|
+| `KASTEN_ENDPOINT` | K10 gateway URL, e.g. `http://gateway.kasten-io.svc.cluster.local/k10` |
+| `KASTEN_TOKEN` | Service-account bearer token with access to `kasten-io` |
+| `KASTEN_POLICY_NAME` | Policy to trigger (default: `rag-daily-backup`) |
+| `KASTEN_NAMESPACE` | Namespace where K10 is installed (default: `kasten-io`) |
+
+To remove Kasten and its configuration:
+
+```bash
+make uninstall-kasten
+```
+
+> **Note:** the chart assumes a set of pgvector defaults. If your deployment differs, override them on the `make` command line or edit [`deploy/helm/kasten/values.yaml`](deploy/helm/kasten/values.yaml) directly.
+>
+> | Variable | Chart value | Default |
+> |----------|-------------|---------|
+> | `PGVECTOR_SECRET` | `pgvectorSecretName` | `pgvector-secret` |
+> | `PGVECTOR_SERVICE` | `pgvectorServiceName` | `pgvector` |
+> | `PGVECTOR_DATABASE` | `pgvectorDatabase` | `rag_blueprint` |
+> | `PGVECTOR_STATEFULSET` | `pgvectorStatefulSetName` | `pgvector` |
+>
+> ```bash
+> make install-kasten NAMESPACE=llama-stack-rag PGVECTOR_SECRET=my-pgvector-secret
+> ```
+
 ### Local Deployment
 
 For local development and testing, see the [Local Setup Guide](docs/local_setup_guide.md).
 
 ## Tags
 
-* **Product:** OpenShift AI
-* **Use case:** RAG
+* **Product:** OpenShift AI, Veeam Kasten
+* **Use case:** RAG, AI data protection and resilience
 * **Business challenge:** Adopt and scale AI
diff --git a/deploy/helm/Makefile b/deploy/helm/Makefile
@@ -241,6 +241,12 @@ help: ## Show this help message
 	@echo -e "  edit-config   - Edit configuration file"
 	@echo -e "  show-config   - Show current configuration"
 	@echo -e ""
+	@echo -e "$(GREEN)Kasten K10 Data Protection Targets:$(NC)"
+	@echo -e "  install-kasten       - Install Kasten K10 and deploy data protection config"
+	@echo -e "  uninstall-kasten     - Uninstall Kasten K10 and remove data protection config"
+	@echo -e "  kasten-set-profile   - Set the K10 Location Profile for off-cluster storage"
+	@echo -e "  kasten-status        - Show Kasten K10 deployment and policy status"
+	@echo -e ""
 	@echo -e "$(GREEN)Configuration:$(NC)"
 	@echo -e "  NAMESPACE     - Target namespace (required)"
 	@echo -e "  VALUES_FILE   - Configuration file (default: rag-values.yaml)"
@@ -695,3 +701,85 @@ delete-jobs: ## Delete all jobs in the namespace
 	@echo -e "$(BLUE)[INFO]$(NC) Deleting all jobs in namespace $(NAMESPACE)..."
 	@oc delete jobs -n $(NAMESPACE) --all ||:
 	@echo -e "$(GREEN)[SUCCESS]$(NC) Job deletion completed"
+
+# ── Kasten K10 data protection ────────────────────────────────────────────────
+KASTEN_NAMESPACE ?= kasten-io
+KASTEN_CHART := kasten-config
+KASTEN_CHART_DIR := kasten
+
+# Optional overrides for the pgvector values in the kasten-config chart.  Each is
+# passed to helm only when set, so unset variables fall back to the chart defaults
+# in kasten/values.yaml rather than overriding them with an empty string.
+KASTEN_PGVECTOR_ARGS := \
+	$(if $(PGVECTOR_SECRET),--set pgvectorSecretName=$(PGVECTOR_SECRET)) \
+	$(if $(PGVECTOR_SERVICE),--set pgvectorServiceName=$(PGVECTOR_SERVICE)) \
+	$(if $(PGVECTOR_DATABASE),--set pgvectorDatabase=$(PGVECTOR_DATABASE)) \
+	$(if $(PGVECTOR_STATEFULSET),--set pgvectorStatefulSetName=$(PGVECTOR_STATEFULSET))
+
+.PHONY: install-kasten
+install-kasten: check-deps ## Install Veeam Kasten K10 and configure data protection for the RAG application
+	@if [ -z "$(NAMESPACE)" ]; then echo -e "$(RED)[ERROR]$(NC) NAMESPACE is required for install-kasten"; exit 1; fi
+	@echo -e "$(BLUE)[INFO]$(NC) Adding Kasten Helm repository..."
+	@helm repo add kasten https://charts.kasten.io/ 2>/dev/null || true
+	@helm repo update
+	@echo -e "$(BLUE)[INFO]$(NC) Installing Veeam Kasten K10 in namespace $(KASTEN_NAMESPACE)..."
+	@helm upgrade --install k10 kasten/k10 \
+		--namespace $(KASTEN_NAMESPACE) \
+		--create-namespace \
+		--set auth.tokenAuth.enabled=true \
+		--set global.openshift.enabled=true
+	@echo -e "$(BLUE)[INFO]$(NC) Deploying Kanister blueprint and Kasten policy..."
+	@helm upgrade --install $(KASTEN_CHART) $(KASTEN_CHART_DIR) \
+		--namespace $(KASTEN_NAMESPACE) \
+		--set ragNamespace=$(NAMESPACE) \
+		$(KASTEN_PGVECTOR_ARGS)
+	@echo -e "$(GREEN)[SUCCESS]$(NC) Kasten K10 installed successfully."
+	@echo -e ""
+	@echo -e "$(YELLOW)[NEXT STEPS]$(NC)"
+	@echo -e "  1. Retrieve your Kasten dashboard route:"
+	@echo -e "       oc get route k10 -n $(KASTEN_NAMESPACE)"
+	@echo -e "  2. Log in with your OpenShift token and create a Location Profile"
+	@echo -e "     (Settings > Locations) pointing to your backup storage."
+	@echo -e "  3. Associate the profile with the policy:"
+	@echo -e "       make kasten-set-profile NAMESPACE=$(NAMESPACE) LOCATION_PROFILE=<profile-name>"
+	@echo -e ""
+	@echo -e "$(YELLOW)[NOTE]$(NC) If the pgvector defaults do not match your cluster, override them:"
+	@echo -e "       make install-kasten NAMESPACE=$(NAMESPACE) PGVECTOR_SECRET=<secret-name>"
+	@echo -e "  Available overrides: PGVECTOR_SECRET (default: pgvector-secret),"
+	@echo -e "  PGVECTOR_SERVICE (pgvector), PGVECTOR_DATABASE (rag_blueprint),"
+	@echo -e "  PGVECTOR_STATEFULSET (pgvector)"
+
+.PHONY: kasten-set-profile
+kasten-set-profile: ## Set the K10 Location Profile for the RAG backup policy (LOCATION_PROFILE=<name>)
+	@if [ -z "$(LOCATION_PROFILE)" ]; then echo -e "$(RED)[ERROR]$(NC) LOCATION_PROFILE is required"; exit 1; fi
+	@if [ -z "$(NAMESPACE)" ]; then echo -e "$(RED)[ERROR]$(NC) NAMESPACE is required"; exit 1; fi
+	@echo -e "$(BLUE)[INFO]$(NC) Updating Kasten policy location profile to '$(LOCATION_PROFILE)'..."
+	@helm upgrade $(KASTEN_CHART) $(KASTEN_CHART_DIR) \
+		--namespace $(KASTEN_NAMESPACE) \
+		--reuse-values \
+		--set ragNamespace=$(NAMESPACE) \
+		--set locationProfile=$(LOCATION_PROFILE) \
+		$(KASTEN_PGVECTOR_ARGS)
+	@echo -e "$(GREEN)[SUCCESS]$(NC) Location profile updated to: $(LOCATION_PROFILE)"
+
+.PHONY: uninstall-kasten
+uninstall-kasten: ## Uninstall Kasten K10 and remove data protection configuration
+	@echo -e "$(BLUE)[INFO]$(NC) Removing Kasten data protection configuration..."
+	@helm -n $(KASTEN_NAMESPACE) uninstall $(KASTEN_CHART) 2>/dev/null || true
+	@echo -e "$(BLUE)[INFO]$(NC) Uninstalling Veeam Kasten K10..."
+	@helm -n $(KASTEN_NAMESPACE) uninstall k10 2>/dev/null || true
+	@echo -e "$(GREEN)[SUCCESS]$(NC) Kasten K10 uninstalled"
+
+.PHONY: kasten-status
+kasten-status: ## Show Kasten K10 deployment and policy status
+	@echo -e "$(BLUE)[INFO]$(NC) Kasten K10 pods in $(KASTEN_NAMESPACE):"
+	@oc get pods -n $(KASTEN_NAMESPACE) 2>/dev/null || true
+	@echo -e ""
+	@echo -e "$(BLUE)[INFO]$(NC) Kasten dashboard route:"
+	@oc get route k10 -n $(KASTEN_NAMESPACE) 2>/dev/null || true
+	@echo -e ""
+	@echo -e "$(BLUE)[INFO]$(NC) Kasten backup policies:"
+	@oc get policies.config.kio.kasten.io -n $(KASTEN_NAMESPACE) 2>/dev/null || true
+	@echo -e ""
+	@echo -e "$(BLUE)[INFO]$(NC) Kanister blueprints:"
+	@oc get blueprints.cr.kanister.io -n $(KASTEN_NAMESPACE) 2>/dev/null || true
diff --git a/deploy/helm/kasten/Chart.yaml b/deploy/helm/kasten/Chart.yaml
@@ -0,0 +1,6 @@
+apiVersion: v2
+name: kasten-config
+description: Kasten K10 data protection configuration for the RAG AI application — deploys a Kanister blueprint for pgvector logical backup and a daily Kasten policy covering the RAG namespace.
+type: application
+version: 0.1.0
+appVersion: "7.0.0"