diff --git a/README.md b/README.md index 9aeca127..22561c6a 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -# Centralize company knowledge with an Enterprise RAG Chatbot +# Centralize company knowledge with an Enterprise RAG Chatbot and Protect it with Veeam Kasten [![GitHub release](https://img.shields.io/github/v/release/rh-ai-quickstart/RAG)](https://github.com/rh-ai-quickstart/RAG/releases) [![Quay.io](https://img.shields.io/badge/quay.io-llamastack--dist--ui-blue)](https://quay.io/repository/rh-ai-quickstart/llamastack-dist-ui) @@ -9,6 +9,7 @@ Use retrieval-augmented generation (RAG) to enhance large language models with s ## Table of Contents - [Detailed description](#detailed-description) + - [Data protection and resilience with Veeam Kasten](#data-protection-and-resilience-with-veeam-kasten) - [Architecture diagrams](#architecture-diagrams) - [Requirements](#requirements) - [Minimum hardware requirements](#minimum-hardware-requirements) @@ -18,6 +19,7 @@ Use retrieval-augmented generation (RAG) to enhance large language models with s - [Prerequisites](#prerequisites) - [Supported Models](#supported-models) - [Installation Steps](#installation-steps) + - [Protect the Deployment with Veeam Kasten](#protect-the-deployment-with-veeam-kasten) - [Local Deployment](#local-deployment) - [Tags](#tags) @@ -37,11 +39,62 @@ This QuickStart allows users to explore the capabilities of RAG by: - Switching between simple and agent based RAG +### Data protection and resilience with Veeam Kasten + +A production RAG chatbot is not a stateless application. The intelligence that makes FantaCo's assistant useful lives in state that is scattered across the namespace: vector embeddings and their relational metadata in PostgreSQL + PGVector, ingested source documents in S3/MinIO object storage, LlamaStack configuration and session data, Kubeflow pipeline definitions and run history, model-serving configuration, and the ConfigMaps, Secrets, and Routes that wire it all together. Lose any one of those pieces and the chatbot does not simply stop — it keeps answering, but with stale, incomplete, or wrong context. Rebuilding it from scratch means re-running ingestion and re-embedding the entire corpus, consuming hours of GPU/CPU time while the business waits. + +**[Veeam Kasten](https://www.veeam.com/products/cloud/kubernetes-data-protection.html) is a Kubernetes-native data management platform** built specifically for containerized applications. Rather than treating a cluster as a set of volumes to be snapshotted, Kasten understands the *application*: it discovers the workloads, custom resources, configuration, and persistent data that constitute a running app, then captures, moves, and restores them as a coherent unit. This QuickStart ships a complete Kasten configuration so you can experience that model against a real AI workload — see [Protect the Deployment with Veeam Kasten](#protect-the-deployment-with-veeam-kasten). + +![How Veeam Kasten protects the RAG AI application on OpenShift](docs/images/kasten-rag-protection.png) + +*Kasten K10 runs in its own `kasten-io` namespace, alongside — not inside — the RAG application: the `rag-daily-backup` policy selects the RAG namespace, snapshots every PVC through CSI, runs the Kanister blueprint against pgvector, and exports the resulting restore point to the location profile you define. Editable source: [`kasten-rag-protection.svg`](docs/images/kasten-rag-protection.svg).* + +#### One application, one restore point + +Kasten captures the entire OpenShift AI RAG application — the vector database, the relational metadata that indexes it, the object-store document corpus, and every Kubernetes object that defines the deployment — into a **single, application-consistent restore point**. + +| RAG component | State protected by Kasten | +|---------------|---------------------------| +| PostgreSQL + PGVector | Vector embeddings, collections, and relational schema and metadata | +| S3 / MinIO document store | Source documents that feed the ingestion pipeline | +| LlamaStack | Configuration, registered models and shields, session state | +| Kubeflow / Data Science Pipelines | Pipeline definitions, run history, and artifacts | +| Model serving (vLLM / KServe) | InferenceService and serving-runtime definitions | +| Application plumbing | Deployments, StatefulSets, Services, Routes, ConfigMaps, Secrets, RBAC, CRs | + +The distinction matters. Backing up the vector database on its own leaves you with embeddings that no longer match the documents they were derived from, or a database that no application knows how to reach. A Kasten restore point is internally consistent across *all* of it, so a recovery returns a working chatbot rather than a pile of correlated-but-unaligned data that an engineer has to reassemble by hand. + +#### Orchestration, not just snapshots + +Stateful services need more than a volume copy. Kasten's orchestration engine — built on the open-source [Kanister](https://kanister.io/) framework — lets each workload define exactly *how* it should be quiesced, captured, and brought back: + +- **Application-aware capture.** The [`pgvector-logical-backup` blueprint](deploy/helm/kasten/templates/pgvector-blueprint.yaml) included here runs a `pg_dump` against the live pgvector instance and streams it directly to your backup target. Because the result is a logical dump rather than a block-level snapshot, it restores cleanly across different storage classes, clusters, and even PostgreSQL versions — the foundation for migration and dev/test cloning, not only recovery. +- **Automatic discovery.** Kasten finds the workloads that need special handling through a single annotation (`kanister.kasten.io/blueprint`), applied to the pgvector StatefulSet automatically at install time. Adding protection for another stateful service is a blueprint and an annotation — not a new backup pipeline. +- **Sequenced, dependency-aware recovery.** On restore, Kasten replays the operation in the correct order: recreate namespace resources, restore persistent volumes, then run the blueprint's logical restore phase, so pods come back attached to the right data with no manual sequencing by an operator at 2 a.m. +- **Policy-driven and declarative.** The [backup policy](deploy/helm/kasten/templates/rag-policy.yaml) shipped here protects everything in the RAG namespace on a schedule with tiered retention (7 daily / 4 weekly / 12 monthly / 5 yearly), exported off-cluster to a location profile. It is a Kubernetes custom resource delivered by Helm, so protection is versioned and deployed alongside the application itself and fits naturally into a GitOps workflow. +- **Programmable, event-driven backups.** Kasten's API can be invoked from anywhere in your automation. This QuickStart wires it into the [data ingestion pipeline](notebooks/data-ingestion-pipeline.ipynb): a `trigger_kasten_backup` step calls the K10 `RunAction` API and blocks until the backup completes *before* any new documents mutate the vector database. Every ingestion run is therefore preceded by a known-good rollback point — so a bad document batch, a corrupted embedding run, or a poisoned source corpus is an undo operation rather than an incident. + +#### Lower RTO, lower operational overhead + +Without a single restore point, recovering this application is a multi-team, multi-hour project: re-provision the namespace, restore or rebuild PostgreSQL, re-seed object storage, re-run ingestion and re-embed the corpus, reconcile secrets and routes, then validate that retrieval quality actually came back. Every step is manual, ordered, and easy to get wrong under pressure. + +With Kasten it is one restore action against one restore point: + +- **RTO measured in minutes, not hours or days.** Recovery becomes a single orchestrated operation instead of a sequence of hand-run procedures — and because the embeddings are restored rather than regenerated, you skip the re-ingestion and re-embedding compute entirely. +- **No bespoke backup tooling to maintain.** One policy replaces per-component scripts, cron jobs, and tribal knowledge. Protection is declared once, in the same Helm workflow that deploys the application. +- **Predictable, testable recovery.** Restores can be rehearsed into a separate namespace or cluster on demand, turning DR from an assumption into something you have actually verified. +- **Reuse beyond recovery.** The same restore point powers cluster migration, environment cloning for evaluation and testing, and ransomware resilience through immutable, off-cluster backup copies. + +For an AI platform team, the practical result is that the RAG application's most valuable and most expensive-to-rebuild asset — its knowledge — is protected with the same rigor, and the same automation, as the code that serves it. + + ### Architecture diagrams ![RAG System Architecture](docs/images/rag-architecture.png) *This diagram illustrates both the ingestion pipeline for document processing and the RAG pipeline for query handling. For more details click [here](docs/rag-reference-architecture.md).* +For how this deployment is backed up, restored, and moved between clusters, see the data protection architecture in [Data protection and resilience with Veeam Kasten](#data-protection-and-resilience-with-veeam-kasten). + | Layer/Component | Technology | Purpose/Description | |-----------------|------------|---------------------| | **Orchestration** | OpenShift AI | Container orchestration and GPU acceleration | @@ -55,6 +108,7 @@ This QuickStart allows users to explore the capabilities of RAG by: | **Retrieval** | Vector Search | Retrieves relevant documents based on query similarity | | **Data Ingestion** | Kubeflow Pipelines | Multi-modal data ingestion with preprocessing pipelines for cleaning, chunking, and embedding generation | | **Storage** | S3 Bucket | Document source for enterprise content | +| **Data Protection** | Veeam Kasten | Application-consistent backup, restore, and mobility for the entire RAG namespace, including the pgvector database | ## Requirements @@ -71,6 +125,7 @@ This QuickStart allows users to explore the capabilities of RAG by: - OpenShift Cluster 4.18+ - OpenShift AI - Helm CLI - helm +- Veeam Kasten (optional, for data protection) - a CSI driver with `VolumeSnapshot` support and an off-cluster object store for backup exports ### Required user permissions - Regular user permission for default deployment @@ -308,12 +363,78 @@ oc get routes -n llama-stack-rag For detailed post-installation verification, configuration options, and usage instructions, see the [complete OpenShift deployment guide](docs/openshift_setup_guide.md). +### Protect the Deployment with Veeam Kasten + +Once the RAG application is running, deploy Veeam Kasten to protect it. See [Data protection and resilience with Veeam Kasten](#data-protection-and-resilience-with-veeam-kasten) for why this matters and for an architecture diagram of how the pieces fit together. The chart in [`deploy/helm/kasten`](deploy/helm/kasten) installs Kasten K10, the pgvector Kanister blueprint, and a daily backup policy covering the whole RAG namespace. + +1. **Install Kasten K10 and the RAG data protection configuration** + +From the `deploy/helm` directory: + +```bash +make install-kasten NAMESPACE=llama-stack-rag +``` + +This installs Kasten K10 into the `kasten-io` namespace with OpenShift token authentication, deploys the `pgvector-logical-backup` Kanister blueprint, applies the `kanister.kasten.io/blueprint` annotation to the pgvector StatefulSet, and creates the `rag-daily-backup` policy. + +2. **Create a Location Profile** + +Open the Kasten dashboard and log in with your OpenShift token: + +```bash +oc get route k10 -n kasten-io +``` + +Under **Settings > Locations**, create a location profile pointing at your off-cluster backup storage (S3, Azure Blob, GCS, NFS, or similar). + +3. **Bind the profile to the backup policy** + +```bash +make kasten-set-profile NAMESPACE=llama-stack-rag LOCATION_PROFILE= +``` + +4. **Verify protection** + +```bash +make kasten-status +``` + +This shows the K10 pods, the dashboard route, the backup policies, and the registered Kanister blueprints. Trigger the policy from the dashboard to create your first restore point and confirm the pgvector blueprint phase runs. + +**Optional: back up before every ingestion run.** The [data ingestion pipeline](notebooks/data-ingestion-pipeline.ipynb) can trigger a Kasten backup before it writes to the vector database. Set the following environment variables before running the notebook; if they are unset, the step is a no-op and the pipeline behaves as before. + +| Variable | Description | +|----------|-------------| +| `KASTEN_ENDPOINT` | K10 gateway URL, e.g. `http://gateway.kasten-io.svc.cluster.local/k10` | +| `KASTEN_TOKEN` | Service-account bearer token with access to `kasten-io` | +| `KASTEN_POLICY_NAME` | Policy to trigger (default: `rag-daily-backup`) | +| `KASTEN_NAMESPACE` | Namespace where K10 is installed (default: `kasten-io`) | + +To remove Kasten and its configuration: + +```bash +make uninstall-kasten +``` + +> **Note:** the chart assumes a set of pgvector defaults. If your deployment differs, override them on the `make` command line or edit [`deploy/helm/kasten/values.yaml`](deploy/helm/kasten/values.yaml) directly. +> +> | Variable | Chart value | Default | +> |----------|-------------|---------| +> | `PGVECTOR_SECRET` | `pgvectorSecretName` | `pgvector-secret` | +> | `PGVECTOR_SERVICE` | `pgvectorServiceName` | `pgvector` | +> | `PGVECTOR_DATABASE` | `pgvectorDatabase` | `rag_blueprint` | +> | `PGVECTOR_STATEFULSET` | `pgvectorStatefulSetName` | `pgvector` | +> +> ```bash +> make install-kasten NAMESPACE=llama-stack-rag PGVECTOR_SECRET=my-pgvector-secret +> ``` + ### Local Deployment For local development and testing, see the [Local Setup Guide](docs/local_setup_guide.md). ## Tags -* **Product:** OpenShift AI -* **Use case:** RAG +* **Product:** OpenShift AI, Veeam Kasten +* **Use case:** RAG, AI data protection and resilience * **Business challenge:** Adopt and scale AI diff --git a/deploy/helm/Makefile b/deploy/helm/Makefile index eef2ddc7..1c9c72c9 100644 --- a/deploy/helm/Makefile +++ b/deploy/helm/Makefile @@ -241,6 +241,12 @@ help: ## Show this help message @echo -e " edit-config - Edit configuration file" @echo -e " show-config - Show current configuration" @echo -e "" + @echo -e "$(GREEN)Kasten K10 Data Protection Targets:$(NC)" + @echo -e " install-kasten - Install Kasten K10 and deploy data protection config" + @echo -e " uninstall-kasten - Uninstall Kasten K10 and remove data protection config" + @echo -e " kasten-set-profile - Set the K10 Location Profile for off-cluster storage" + @echo -e " kasten-status - Show Kasten K10 deployment and policy status" + @echo -e "" @echo -e "$(GREEN)Configuration:$(NC)" @echo -e " NAMESPACE - Target namespace (required)" @echo -e " VALUES_FILE - Configuration file (default: rag-values.yaml)" @@ -695,3 +701,85 @@ delete-jobs: ## Delete all jobs in the namespace @echo -e "$(BLUE)[INFO]$(NC) Deleting all jobs in namespace $(NAMESPACE)..." @oc delete jobs -n $(NAMESPACE) --all ||: @echo -e "$(GREEN)[SUCCESS]$(NC) Job deletion completed" + +# ── Kasten K10 data protection ──────────────────────────────────────────────── +KASTEN_NAMESPACE ?= kasten-io +KASTEN_CHART := kasten-config +KASTEN_CHART_DIR := kasten + +# Optional overrides for the pgvector values in the kasten-config chart. Each is +# passed to helm only when set, so unset variables fall back to the chart defaults +# in kasten/values.yaml rather than overriding them with an empty string. +KASTEN_PGVECTOR_ARGS := \ + $(if $(PGVECTOR_SECRET),--set pgvectorSecretName=$(PGVECTOR_SECRET)) \ + $(if $(PGVECTOR_SERVICE),--set pgvectorServiceName=$(PGVECTOR_SERVICE)) \ + $(if $(PGVECTOR_DATABASE),--set pgvectorDatabase=$(PGVECTOR_DATABASE)) \ + $(if $(PGVECTOR_STATEFULSET),--set pgvectorStatefulSetName=$(PGVECTOR_STATEFULSET)) + +.PHONY: install-kasten +install-kasten: check-deps ## Install Veeam Kasten K10 and configure data protection for the RAG application + @if [ -z "$(NAMESPACE)" ]; then echo -e "$(RED)[ERROR]$(NC) NAMESPACE is required for install-kasten"; exit 1; fi + @echo -e "$(BLUE)[INFO]$(NC) Adding Kasten Helm repository..." + @helm repo add kasten https://charts.kasten.io/ 2>/dev/null || true + @helm repo update + @echo -e "$(BLUE)[INFO]$(NC) Installing Veeam Kasten K10 in namespace $(KASTEN_NAMESPACE)..." + @helm upgrade --install k10 kasten/k10 \ + --namespace $(KASTEN_NAMESPACE) \ + --create-namespace \ + --set auth.tokenAuth.enabled=true \ + --set global.openshift.enabled=true + @echo -e "$(BLUE)[INFO]$(NC) Deploying Kanister blueprint and Kasten policy..." + @helm upgrade --install $(KASTEN_CHART) $(KASTEN_CHART_DIR) \ + --namespace $(KASTEN_NAMESPACE) \ + --set ragNamespace=$(NAMESPACE) \ + $(KASTEN_PGVECTOR_ARGS) + @echo -e "$(GREEN)[SUCCESS]$(NC) Kasten K10 installed successfully." + @echo -e "" + @echo -e "$(YELLOW)[NEXT STEPS]$(NC)" + @echo -e " 1. Retrieve your Kasten dashboard route:" + @echo -e " oc get route k10 -n $(KASTEN_NAMESPACE)" + @echo -e " 2. Log in with your OpenShift token and create a Location Profile" + @echo -e " (Settings > Locations) pointing to your backup storage." + @echo -e " 3. Associate the profile with the policy:" + @echo -e " make kasten-set-profile NAMESPACE=$(NAMESPACE) LOCATION_PROFILE=" + @echo -e "" + @echo -e "$(YELLOW)[NOTE]$(NC) If the pgvector defaults do not match your cluster, override them:" + @echo -e " make install-kasten NAMESPACE=$(NAMESPACE) PGVECTOR_SECRET=" + @echo -e " Available overrides: PGVECTOR_SECRET (default: pgvector-secret)," + @echo -e " PGVECTOR_SERVICE (pgvector), PGVECTOR_DATABASE (rag_blueprint)," + @echo -e " PGVECTOR_STATEFULSET (pgvector)" + +.PHONY: kasten-set-profile +kasten-set-profile: ## Set the K10 Location Profile for the RAG backup policy (LOCATION_PROFILE=) + @if [ -z "$(LOCATION_PROFILE)" ]; then echo -e "$(RED)[ERROR]$(NC) LOCATION_PROFILE is required"; exit 1; fi + @if [ -z "$(NAMESPACE)" ]; then echo -e "$(RED)[ERROR]$(NC) NAMESPACE is required"; exit 1; fi + @echo -e "$(BLUE)[INFO]$(NC) Updating Kasten policy location profile to '$(LOCATION_PROFILE)'..." + @helm upgrade $(KASTEN_CHART) $(KASTEN_CHART_DIR) \ + --namespace $(KASTEN_NAMESPACE) \ + --reuse-values \ + --set ragNamespace=$(NAMESPACE) \ + --set locationProfile=$(LOCATION_PROFILE) \ + $(KASTEN_PGVECTOR_ARGS) + @echo -e "$(GREEN)[SUCCESS]$(NC) Location profile updated to: $(LOCATION_PROFILE)" + +.PHONY: uninstall-kasten +uninstall-kasten: ## Uninstall Kasten K10 and remove data protection configuration + @echo -e "$(BLUE)[INFO]$(NC) Removing Kasten data protection configuration..." + @helm -n $(KASTEN_NAMESPACE) uninstall $(KASTEN_CHART) 2>/dev/null || true + @echo -e "$(BLUE)[INFO]$(NC) Uninstalling Veeam Kasten K10..." + @helm -n $(KASTEN_NAMESPACE) uninstall k10 2>/dev/null || true + @echo -e "$(GREEN)[SUCCESS]$(NC) Kasten K10 uninstalled" + +.PHONY: kasten-status +kasten-status: ## Show Kasten K10 deployment and policy status + @echo -e "$(BLUE)[INFO]$(NC) Kasten K10 pods in $(KASTEN_NAMESPACE):" + @oc get pods -n $(KASTEN_NAMESPACE) 2>/dev/null || true + @echo -e "" + @echo -e "$(BLUE)[INFO]$(NC) Kasten dashboard route:" + @oc get route k10 -n $(KASTEN_NAMESPACE) 2>/dev/null || true + @echo -e "" + @echo -e "$(BLUE)[INFO]$(NC) Kasten backup policies:" + @oc get policies.config.kio.kasten.io -n $(KASTEN_NAMESPACE) 2>/dev/null || true + @echo -e "" + @echo -e "$(BLUE)[INFO]$(NC) Kanister blueprints:" + @oc get blueprints.cr.kanister.io -n $(KASTEN_NAMESPACE) 2>/dev/null || true diff --git a/deploy/helm/kasten/Chart.yaml b/deploy/helm/kasten/Chart.yaml new file mode 100644 index 00000000..f657bc1f --- /dev/null +++ b/deploy/helm/kasten/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: kasten-config +description: Kasten K10 data protection configuration for the RAG AI application — deploys a Kanister blueprint for pgvector logical backup and a daily Kasten policy covering the RAG namespace. +type: application +version: 0.1.0 +appVersion: "7.0.0" diff --git a/deploy/helm/kasten/templates/_helpers.tpl b/deploy/helm/kasten/templates/_helpers.tpl new file mode 100644 index 00000000..0b02fc3c --- /dev/null +++ b/deploy/helm/kasten/templates/_helpers.tpl @@ -0,0 +1,16 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "kasten-config.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels. +*/}} +{{- define "kasten-config.labels" -}} +helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }} +app.kubernetes.io/name: {{ include "kasten-config.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} diff --git a/deploy/helm/kasten/templates/annotate-pgvector.yaml b/deploy/helm/kasten/templates/annotate-pgvector.yaml new file mode 100644 index 00000000..b819032f --- /dev/null +++ b/deploy/helm/kasten/templates/annotate-pgvector.yaml @@ -0,0 +1,94 @@ +# RBAC + Job — annotate the pgvector StatefulSet for Kanister discovery +# +# Kasten K10 discovers which workloads require Kanister-based logical backup +# through the annotation: +# kanister.kasten.io/blueprint: +# +# Since the pgvector StatefulSet is managed by the RAG Helm release (a separate +# chart), we annotate it via a post-install/post-upgrade Helm hook Job that +# runs `kubectl annotate` after this chart is deployed. +# +# The Job runs in the kasten-io namespace using a dedicated ServiceAccount that +# is granted the minimum RBAC required to patch StatefulSets in the RAG namespace. +--- +# ServiceAccount used by the annotation Job +apiVersion: v1 +kind: ServiceAccount +metadata: + name: kasten-pgvector-annotator + namespace: {{ .Values.kastenNamespace }} + labels: + {{- include "kasten-config.labels" . | nindent 4 }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "-5" + helm.sh/hook-delete-policy: before-hook-creation +--- +# ClusterRole — allows patching StatefulSets (used only in the RAG namespace +# via the RoleBinding below; kept as ClusterRole for re-use across namespaces) +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: kasten-statefulset-annotator + labels: + {{- include "kasten-config.labels" . | nindent 4 }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "-5" + helm.sh/hook-delete-policy: before-hook-creation +rules: +- apiGroups: ["apps"] + resources: ["statefulsets"] + verbs: ["get", "patch"] +--- +# RoleBinding — scopes the ClusterRole to the RAG namespace only +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: kasten-pgvector-annotator + namespace: {{ .Values.ragNamespace }} + labels: + {{- include "kasten-config.labels" . | nindent 4 }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "-5" + helm.sh/hook-delete-policy: before-hook-creation +subjects: +- kind: ServiceAccount + name: kasten-pgvector-annotator + namespace: {{ .Values.kastenNamespace }} +roleRef: + kind: ClusterRole + name: kasten-statefulset-annotator + apiGroup: rbac.authorization.k8s.io +--- +# Job — adds the Kanister blueprint annotation to the pgvector StatefulSet +apiVersion: batch/v1 +kind: Job +metadata: + name: kasten-annotate-pgvector + namespace: {{ .Values.kastenNamespace }} + labels: + {{- include "kasten-config.labels" . | nindent 4 }} + annotations: + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +spec: + ttlSecondsAfterFinished: 300 + template: + spec: + serviceAccountName: kasten-pgvector-annotator + restartPolicy: OnFailure + containers: + - name: annotator + image: bitnami/kubectl:latest + command: + - kubectl + - annotate + - statefulset + - {{ .Values.pgvectorStatefulSetName }} + - -n + - {{ .Values.ragNamespace }} + - --overwrite + - kanister.kasten.io/blueprint={{ .Values.blueprintName }} diff --git a/deploy/helm/kasten/templates/pgvector-blueprint.yaml b/deploy/helm/kasten/templates/pgvector-blueprint.yaml new file mode 100644 index 00000000..7475d864 --- /dev/null +++ b/deploy/helm/kasten/templates/pgvector-blueprint.yaml @@ -0,0 +1,117 @@ +# Kanister Blueprint — pgvector logical backup and restore +# +# Performs a pg_dump / pg_restore against the pgvector PostgreSQL instance. +# The blueprint is triggered automatically by Kasten K10 whenever it backs up +# a StatefulSet that carries the annotation: +# kanister.kasten.io/blueprint: {{ .Values.blueprintName }} +# +# The annotation is applied to the pgvector StatefulSet by the Job in +# annotate-pgvector.yaml during chart installation. +# +# Blueprint image: ghcr.io/kanisterio/postgresql +# Contains pg_dump, pg_restore, and the `kando` CLI (for location profile I/O). +# +# NOTE: Kanister uses its own Go template engine at run-time. The {{ "{{" }} … {{ "}}" }} +# expressions below are Kanister runtime templates, not Helm templates. +apiVersion: cr.kanister.io/v1alpha1 +kind: Blueprint +metadata: + name: {{ .Values.blueprintName }} + namespace: {{ .Values.kastenNamespace }} + labels: + {{- include "kasten-config.labels" . | nindent 4 }} +actions: + # ── backup ────────────────────────────────────────────────────────────────── + backup: + kind: StatefulSet + outputArtifacts: + pgvectorDump: + keyValue: + path: '{{ "{{" }} .Phases.pgDump.Output.path {{ "}}" }}' + phases: + - func: KubeTask + name: pgDump + objects: + pgvectorSecret: + kind: Secret + name: {{ .Values.pgvectorSecretName }} + namespace: '{{ "{{" }} .StatefulSet.Namespace {{ "}}" }}' + args: + image: ghcr.io/kanisterio/postgresql:0.109.0 + namespace: '{{ "{{" }} .StatefulSet.Namespace {{ "}}" }}' + command: + - bash + - -o + - errexit + - -o + - pipefail + - -c + - | + backup_path="pgvector-backups/{{ "{{" }} .StatefulSet.Namespace {{ "}}" }}/{{ "{{" }} toDate "2006-01-02T15:04:05.999999999Z07:00" .Time | date "2006-01-02T15-04-05" {{ "}}" }}/pgvector-dump.pgdump" + PGPASSWORD="{{ "{{" }} index .Phases.pgDump.Secrets.pgvectorSecret.Data "password" | toString {{ "}}" }}" \ + pg_dump \ + -h {{ .Values.pgvectorServiceName }} \ + -U "{{ "{{" }} index .Phases.pgDump.Secrets.pgvectorSecret.Data "user" | toString {{ "}}" }}" \ + -d {{ .Values.pgvectorDatabase }} \ + -Fc | \ + kando location push \ + --profile '{{ "{{" }} toJson .Profile {{ "}}" }}' \ + --path "${backup_path}" \ + --stdin + kando output path "${backup_path}" + + # ── restore ───────────────────────────────────────────────────────────────── + restore: + kind: StatefulSet + inputArtifactNames: + - pgvectorDump + phases: + - func: KubeTask + name: pgRestore + objects: + pgvectorSecret: + kind: Secret + name: {{ .Values.pgvectorSecretName }} + namespace: '{{ "{{" }} .StatefulSet.Namespace {{ "}}" }}' + args: + image: ghcr.io/kanisterio/postgresql:0.109.0 + namespace: '{{ "{{" }} .StatefulSet.Namespace {{ "}}" }}' + command: + - bash + - -o + - errexit + - -o + - pipefail + - -c + - | + kando location pull \ + --profile '{{ "{{" }} toJson .Profile {{ "}}" }}' \ + --path '{{ "{{" }} .ArtifactsIn.pgvectorDump.KeyValue.path {{ "}}" }}' \ + --stdout | \ + PGPASSWORD="{{ "{{" }} index .Phases.pgRestore.Secrets.pgvectorSecret.Data "password" | toString {{ "}}" }}" \ + pg_restore \ + -h {{ .Values.pgvectorServiceName }} \ + -U "{{ "{{" }} index .Phases.pgRestore.Secrets.pgvectorSecret.Data "user" | toString {{ "}}" }}" \ + -d {{ .Values.pgvectorDatabase }} \ + --clean --if-exists + + # ── delete ────────────────────────────────────────────────────────────────── + delete: + type: Namespaced + phases: + - func: KubeTask + name: deleteBackup + args: + image: ghcr.io/kanisterio/postgresql:0.109.0 + namespace: '{{ "{{" }} .Namespace.Name {{ "}}" }}' + command: + - bash + - -o + - errexit + - -o + - pipefail + - -c + - | + kando location delete \ + --profile '{{ "{{" }} toJson .Profile {{ "}}" }}' \ + --path '{{ "{{" }} .ArtifactsIn.pgvectorDump.KeyValue.path {{ "}}" }}' diff --git a/deploy/helm/kasten/templates/rag-policy.yaml b/deploy/helm/kasten/templates/rag-policy.yaml new file mode 100644 index 00000000..58ce38dc --- /dev/null +++ b/deploy/helm/kasten/templates/rag-policy.yaml @@ -0,0 +1,43 @@ +# Kasten K10 Policy — daily backup of the RAG AI application namespace +# +# Selects every application in .Values.ragNamespace and backs it up on the +# configured schedule. The pgvector StatefulSet is annotated with the Kanister +# blueprint name, so Kasten will invoke the logical pg_dump blueprint in +# addition to the standard volume snapshot for all other PVCs. +# +# Prerequisites: +# - A K10 Location Profile must exist in the kasten-io namespace before the +# policy can export data off-cluster. Set .Values.locationProfile to the +# profile name (or update it later with: make kasten-set-profile). +apiVersion: config.kio.kasten.io/v1alpha1 +kind: Policy +metadata: + name: {{ .Values.policy.name }} + namespace: {{ .Values.kastenNamespace }} + labels: + {{- include "kasten-config.labels" . | nindent 4 }} +spec: + comment: "Daily backup of the RAG AI application and pgvector database in namespace {{ .Values.ragNamespace }}" + frequency: {{ .Values.policy.frequency | quote }} + paused: false + selector: + matchExpressions: + - key: k10.kasten.io/appNamespace + operator: In + values: + - {{ .Values.ragNamespace | quote }} + actions: + - action: backup + backupParameters: + {{- if .Values.locationProfile }} + profile: + clusterName: "" + name: {{ .Values.locationProfile | quote }} + namespace: {{ .Values.kastenNamespace | quote }} + {{- end }} + filters: {} + retention: + daily: {{ .Values.policy.retention.daily }} + weekly: {{ .Values.policy.retention.weekly }} + monthly: {{ .Values.policy.retention.monthly }} + yearly: {{ .Values.policy.retention.yearly }} diff --git a/deploy/helm/kasten/values.yaml b/deploy/helm/kasten/values.yaml new file mode 100644 index 00000000..676f126f --- /dev/null +++ b/deploy/helm/kasten/values.yaml @@ -0,0 +1,45 @@ +# Target namespace of the RAG application to protect. +# Should match the NAMESPACE used when installing the RAG chart. +ragNamespace: llama-stack-rag + +# Kasten K10 namespace — must match the namespace K10 was installed into. +kastenNamespace: kasten-io + +# Name of the K10 Location Profile to use for off-cluster backup storage. +# Create the profile in the Kasten dashboard (Settings > Locations) before +# or shortly after chart installation, then update this value with: +# make kasten-set-profile LOCATION_PROFILE= NAMESPACE= +# Leave empty to deploy the policy without a location binding (policy will +# remain inactive until a profile is assigned). +locationProfile: "" + +# Name of the Kubernetes Secret that holds pgvector credentials. +# This secret is created by the pgvector subchart in the RAG namespace. +# Typical name: pgvector-secret (adjust if your chart uses a different convention) +pgvectorSecretName: pgvector-secret + +# Kubernetes Service name of the pgvector instance. +# Matches the service name created by the pgvector subchart. +pgvectorServiceName: pgvector + +# PostgreSQL database name to dump/restore. +pgvectorDatabase: rag_blueprint + +# Name of the pgvector StatefulSet — used by the annotation job. +pgvectorStatefulSetName: pgvector + +# Name assigned to the Kanister Blueprint resource. +blueprintName: pgvector-logical-backup + +# Kasten K10 backup policy settings. +policy: + # Name of the K10 Policy resource. + name: rag-daily-backup + # Backup schedule. Accepts cron syntax or shorthand: @daily, @weekly, @hourly. + frequency: "@daily" + # Retention settings (number of restore points to keep per period). + retention: + daily: 7 + weekly: 4 + monthly: 12 + yearly: 5 diff --git a/docs/images/kasten-rag-protection.png b/docs/images/kasten-rag-protection.png new file mode 100644 index 00000000..fc2695b7 Binary files /dev/null and b/docs/images/kasten-rag-protection.png differ diff --git a/docs/images/kasten-rag-protection.svg b/docs/images/kasten-rag-protection.svg new file mode 100644 index 00000000..12289cbc --- /dev/null +++ b/docs/images/kasten-rag-protection.svg @@ -0,0 +1,190 @@ + + Veeam Kasten protecting the OpenShift AI RAG application + + + + + + + + + + + + + + Veeam Kasten protects the RAG AI application as a single application + K10 runs beside the workload in the same cluster, captures its state as a single restore point, and exports that point off-cluster. + + + + OPENSHIFT CONTAINER PLATFORM 4.18+ · OPENSHIFT AI + + + + namespace: llama-stack-rag · RAG AI application + Stateless runtime — recreated from the captured manifests, not from a rebuild + + + + Streamlit chat UI + OpenShift Route · :8501 + chat · upload · inspect + + + LlamaStack API + :8321 · agents, shields + in-process embeddings + + + vLLM on KServe + Llama-3.2-3B · Guard-3-8B + GPU · weights on emptyDir + + + Ingestion + MCP + rag-ingestion-pipeline + DSPA · MCP tool servers + + + + Application state — the expensive-to-rebuild part, captured together as ONE application-consistent restore point + + + pgvector StatefulSet + PostgreSQL 17 + pgvector + embeddings, collections, + relational metadata + pg-data-pgvector-0 · 5Gi + + kanister.kasten.io/blueprint + + + MinIO / S3 store + source document corpus + bucket: documents + HR, legal, sales, IT PDFs + minio-data-minio-0 · 50Gi + + + LlamaStack + KFP + registry, sessions, files + pipeline defs, run history + llama-stack-data · 2Gi + mariadb-dspa · 10Gi + pipeline-vol · 1Gi + + + Kubernetes objects + Deployments, StatefulSets + Services, Routes, RBAC + ConfigMaps & Secrets + InferenceService and + ServingRuntime CRs + + + + namespace: kasten-io · Veeam Kasten K10 — same cluster, its own namespace + + + Kanister Blueprint + pgvector-logical-backup + pg_dump -Fc | kando push + logical, not block-level: + restores across storage + classes, clusters and + PostgreSQL versions + + + Backup policy + rag-daily-backup + frequency: @daily + selector: appNamespace + = llama-stack-rag + retain 7d / 4w / 12m / 5y + shipped by Helm as a CR + + + K10 platform + helm install kasten/k10 + dashboard Route + OpenShift token auth + catalog · executor · + gateway + RunAction API + + + Export off-cluster + backupParameters.profile + part of the same backup + action — no second step + you bind the profile with + make kasten-set-profile + deduplicated + encrypted + + + + + pg_dump / + pg_restore + against the live DB + + + + @daily backup of the + whole RAG namespace + CSI VolumeSnapshot + + + + RunAction API + the pipeline backs up + before every ingest run + + + + export + + restore + + + + OFF-CLUSTER BACKUP TARGET + + Location profile + you create this yourself in the K10 + dashboard: Settings › Locations + + + S3 · Azure Blob · GCS · NFS + object-lock immutability optional + + ONE RESTORE POINT CONTAINS + + · Deployments, StatefulSets + · Services, Routes, RBAC + · ConfigMaps, Secrets, CRs + · PVC volume data (CSI) + · pgvector .pgdump (Kanister) + · MinIO document corpus + + RETENTION + + 7 daily · 4 weekly + 12 monthly · 5 yearly + + RECOVERY & MOBILITY + + · Restore the same namespace + · Clone into a new namespace + · Migrate to another cluster + · Rehearse DR without prod risk + · Roll back a bad ingestion run + · Ransomware recovery + + Embeddings come back restored, not + recomputed — no re-ingestion, no GPU + hours, RTO in minutes. + diff --git a/notebooks/data-ingestion-pipeline.ipynb b/notebooks/data-ingestion-pipeline.ipynb index f960da9d..d338ed08 100644 --- a/notebooks/data-ingestion-pipeline.ipynb +++ b/notebooks/data-ingestion-pipeline.ipynb @@ -29,6 +29,97 @@ "- stores the data in Postgres Vector database" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from kfp.dsl import component\n", + "\n", + "\n", + "@component(\n", + " base_image=\"python:3.12\",\n", + " packages_to_install=[\"requests\"]\n", + ")\n", + "def trigger_kasten_backup(\n", + " kasten_endpoint: str,\n", + " kasten_token: str,\n", + " policy_name: str,\n", + " namespace: str = \"kasten-io\",\n", + ") -> str:\n", + " \"\"\"\n", + " Trigger a Kasten K10 backup policy run and wait for completion.\n", + "\n", + " If kasten_endpoint or kasten_token are empty the step is a no-op.\n", + " Returns the RunAction name on success, or an empty string when skipped.\n", + "\n", + " Set the following environment variables before running the pipeline to\n", + " enable pre-ingestion backup:\n", + " KASTEN_ENDPOINT - Kasten K10 gateway URL, e.g. http://gateway.kasten-io.svc.cluster.local/k10\n", + " KASTEN_TOKEN - OpenShift service-account bearer token with access to kasten-io\n", + " KASTEN_POLICY_NAME - Name of the K10 Policy to trigger (default: rag-daily-backup)\n", + " KASTEN_NAMESPACE - Namespace where K10 is installed (default: kasten-io)\n", + " \"\"\"\n", + " import time\n", + " import requests\n", + " import urllib3\n", + " urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)\n", + "\n", + " if not kasten_endpoint or not kasten_token:\n", + " print(\"Kasten endpoint or token not provided \u2014 skipping pre-ingestion backup.\")\n", + " return \"\"\n", + "\n", + " headers = {\n", + " \"Authorization\": f\"Bearer {kasten_token}\",\n", + " \"Content-Type\": \"application/json\",\n", + " }\n", + "\n", + " # Trigger the policy run via the Kasten K10 RunAction API\n", + " run_url = (\n", + " f\"{kasten_endpoint}/apis/actions.kio.kasten.io/v1alpha1\"\n", + " f\"/namespaces/{namespace}/runactions\"\n", + " )\n", + " payload = {\n", + " \"apiVersion\": \"actions.kio.kasten.io/v1alpha1\",\n", + " \"kind\": \"RunAction\",\n", + " \"spec\": {\n", + " \"subject\": {\n", + " \"apiVersion\": \"config.kio.kasten.io/v1alpha1\",\n", + " \"kind\": \"Policy\",\n", + " \"name\": policy_name,\n", + " \"namespace\": namespace,\n", + " }\n", + " },\n", + " }\n", + "\n", + " resp = requests.post(run_url, headers=headers, json=payload, verify=False)\n", + " resp.raise_for_status()\n", + " run_action = resp.json()\n", + " run_name = run_action[\"metadata\"][\"name\"]\n", + " print(f\"Kasten backup policy triggered. RunAction: {run_name}\")\n", + "\n", + " # Poll until the run completes, fails, or times out (30 min max)\n", + " status_url = (\n", + " f\"{kasten_endpoint}/apis/actions.kio.kasten.io/v1alpha1\"\n", + " f\"/namespaces/{namespace}/runactions/{run_name}\"\n", + " )\n", + " deadline = time.time() + 1800\n", + " while time.time() < deadline:\n", + " status_resp = requests.get(status_url, headers=headers, verify=False)\n", + " status_resp.raise_for_status()\n", + " state = status_resp.json().get(\"status\", {}).get(\"state\", \"\")\n", + " print(f\"Backup state: {state}\")\n", + " if state == \"complete\":\n", + " print(\"Kasten backup completed successfully.\")\n", + " return run_name\n", + " if state in (\"failed\", \"cancelled\"):\n", + " raise RuntimeError(f\"Kasten backup ended with state: {state}\")\n", + " time.sleep(30)\n", + "\n", + " raise TimeoutError(\"Kasten backup timed out after 30 minutes.\")\n" + ] + }, { "cell_type": "code", "execution_count": null, @@ -220,16 +311,32 @@ "\n", "\n", "@pipeline(name=\"fetch-docling-process-store-pipeline\")\n", - "def full_pipeline():\n", + "def full_pipeline(\n", + " kasten_endpoint: str = \"\",\n", + " kasten_token: str = \"\",\n", + " kasten_policy_name: str = \"rag-daily-backup\",\n", + " kasten_namespace: str = \"kasten-io\",\n", + "):\n", " import os\n", "\n", + " # Step 0: Trigger a Kasten backup before modifying the vector database.\n", + " # If KASTEN_ENDPOINT / KASTEN_TOKEN are not set the step is a no-op.\n", + " backup_task = trigger_kasten_backup(\n", + " kasten_endpoint=kasten_endpoint,\n", + " kasten_token=kasten_token,\n", + " policy_name=kasten_policy_name,\n", + " namespace=kasten_namespace,\n", + " )\n", + "\n", + " # Step 1-3: Fetch from MinIO, chunk with Docling, store in pgvector.\n", + " # Runs only after the backup step completes (or is skipped).\n", " fetch_from_minio_docling_process_store(\n", " bucket_name=\"documents\",\n", " minio_endpoint=os.environ[\"MINIO_ENDPOINT\"],\n", " minio_access_key=os.environ[\"MINIO_ACCESS_KEY\"],\n", " minio_secret_key=os.environ[\"MINIO_SECRET_KEY\"],\n", " llamastack_base_url=os.environ[\"LLAMASTACK_BASE_URL\"]\n", - " )\n", + " ).after(backup_task)\n", "\n", "# 1. Compile pipeline to a file\n", "pipeline_yaml = \"fetch_docling_process_pipeline.yaml\"\n", @@ -253,7 +360,12 @@ "# 4. Run the pipeline\n", "run = client.create_run_from_pipeline_package(\n", " pipeline_file=pipeline_yaml,\n", - " arguments={},\n", + " arguments={\n", + " \"kasten_endpoint\": os.environ.get(\"KASTEN_ENDPOINT\", \"\"),\n", + " \"kasten_token\": os.environ.get(\"KASTEN_TOKEN\", \"\"),\n", + " \"kasten_policy_name\": os.environ.get(\"KASTEN_POLICY_NAME\", \"rag-daily-backup\"),\n", + " \"kasten_namespace\": os.environ.get(\"KASTEN_NAMESPACE\", \"kasten-io\"),\n", + " },\n", " run_name=\"fetch-docling-process-store-run\"\n", ")\n", "\n", @@ -290,4 +402,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file