Skip to content

docs: serve an llms.txt index of the documentation - #1939

Open
ghostiee-11 wants to merge 7 commits into
holoviz:mainfrom
ghostiee-11:add/llms-txt
Open

docs: serve an llms.txt index of the documentation#1939
ghostiee-11 wants to merge 7 commits into
holoviz:mainfrom
ghostiee-11:add/llms-txt

Conversation

@ghostiee-11

Copy link
Copy Markdown
Collaborator

LLM assistants now handle a large share of the "which library should I use for this" questions that used to go to search, and they read a site far better when it publishes a curated index than when they have to crawl it. llms.txt is the convention for that. marimo and Vizro both serve one. No HoloViz site does.

This adds docs/llms.txt.

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.49%. Comparing base (0e33053) to head (547a70b).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1939      +/-   ##
==========================================
+ Coverage   75.27%   75.49%   +0.21%     
==========================================
  Files         212      214       +2     
  Lines       39238    39606     +368     
==========================================
+ Hits        29538    29899     +361     
- Misses       9700     9707       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ahuang11

Copy link
Copy Markdown
Contributor

Let's wait for https://github.com/holoviz/hvplot/pull/1732/changes to settle

@ahuang11
ahuang11 marked this pull request as draft July 21, 2026 22:41
@ghostiee-11

Copy link
Copy Markdown
Collaborator Author

Okay!!

@ghostiee-11
ghostiee-11 marked this pull request as ready for review July 24, 2026 17:20
@ahuang11

Copy link
Copy Markdown
Contributor

Do you have an example artifact I can look at

@ghostiee-11

Copy link
Copy Markdown
Collaborator Author

lumen-llms.txt

pr1939-markdown-mirror

@ahuang11

ahuang11 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor
> Lumen is an open-source and extensible agent framework for chatting with data and for retrieval augmented generation. It turns natural language into SQL, data transformation pipelines, visualizations and dashboards, and every step stays inspectable, editable and reproducible.

Lumen is built on Panel and the wider HoloViz stack. Its declarative data model means anything the language model produces can be serialized to YAML, reopened in a notebook, or composed into a dashboard.

Install with `pip install 'lumen[ai-openai]'` and start with `lumen-ai serve data.csv`.

Is this necessary for the LLM to know? And is it named lumen-llms.txt by you, or should it be named llms.txt?

These markdown files should help the LLM develop (think of it like a CLAUDE.md file).
For ref: https://panel-material-ui.holoviz.org/llms.txt

@ahuang11

ahuang11 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Let's wait until holoviz-dev/nbsite#361 is merged

@ahuang11
ahuang11 marked this pull request as draft August 4, 2026 16:08
ghostiee-11 and others added 4 commits August 24, 2026 21:55
Zensical copies static files from docs/ to the site root, so this needs no build changes. A test keeps the link list in sync with the pages that actually exist.
Replaces the hand-written docs/llms.txt with a build step, following the
approach in holoviz/hvplot#1732. The zensical nav is the single source of
truth, so pages and labels can never drift from the site, and the linked
markdown is mirrored under /markdown so an LLM can fetch the prose
without parsing the rendered HTML.
The tutorials index links seven tutorials but the nav only carried four,
so the rest were published without a sidebar entry and were skipped by
llms.txt, which is generated from the nav. Labels match the index page.
@ghostiee-11
ghostiee-11 marked this pull request as ready for review August 24, 2026 17:21
@ahuang11

Copy link
Copy Markdown
Contributor

Can you attach the latest llms.txt

@ghostiee-11

Copy link
Copy Markdown
Collaborator Author

Here's the current output:

llms.txt
# Lumen

Lumen is an open-source and extensible agent framework for chatting with data and for retrieval augmented generation. It turns natural language into SQL, data transformation pipelines, visualizations and dashboards, and every step stays inspectable, editable and reproducible.

All documentation is available as markdown files under /markdown/.

## Overview

Quick start, installation, and top-level pages.

- [Extending Lumen](/markdown/extending.md)
- [FAQ](/markdown/faq.md)
- [Home](/markdown/index.md)
- [Installation](/markdown/installation.md)
- [Quick Start](/markdown/quick_start.md)

## Getting Started

Launching Lumen, navigating the UI, and building your first app.

- [Building Lumen Apps](/markdown/getting_started/building_lumen_apps.md)
- [Launching Lumen](/markdown/getting_started/launching_lumen.md)
- [Navigating the UI](/markdown/getting_started/navigating_the_ui.md)
- [Using Lumen AI](/markdown/getting_started/using_lumen_ai.md)

## Examples

### Tutorials
> Full walkthroughs building an AI-driven data exploration app end to end.

- [Census Data AI Explorer](/markdown/examples/tutorials/census_data_ai_explorer.md)
- [Overview](/markdown/examples/tutorials/index.md)
- [Stock Market Explorer](/markdown/examples/tutorials/massive_stock_explorer.md)
- [Mesonet Weather Explorer](/markdown/examples/tutorials/mesonet_weather_explorer.md)
- [Build Dashboard with Spec](/markdown/examples/tutorials/penguins_dashboard_spec.md)
- [SaaS Company Report Dashboard](/markdown/examples/tutorials/saas_company_report_dashboard.md)
- [SaaS Executive Dashboard](/markdown/examples/tutorials/saas_metrics_reports.md)
- [Weather Data AI Explorer](/markdown/examples/tutorials/weather_data_ai_explorer.md)
- [Weather API Explorer (OpenAPI)](/markdown/examples/tutorials/weather_openapi_explorer.md)

### Gallery
> Short example specs demonstrating individual sources, transforms, and views.

- [London Bike Points](/markdown/examples/gallery/bikes.md)
- [Earthquakes](/markdown/examples/gallery/earthquakes.md)
- [Overview](/markdown/examples/gallery/index.md)
- [NYC Taxi](/markdown/examples/gallery/nyc_taxi.md)
- [Penguins](/markdown/examples/gallery/penguins.md)
- [Precipitation](/markdown/examples/gallery/precip.md)
- [Seattle Weather](/markdown/examples/gallery/seattle.md)
- [Wind Turbines](/markdown/examples/gallery/windturbines.md)

## Configuration

Top-level spec reference: sources, transforms, views, agents, and more.

- [Agents](/markdown/configuration/agents.md)
- [Analyses](/markdown/configuration/analyses.md)
- [CLI](/markdown/configuration/cli.md)
- [Context](/markdown/configuration/context.md)
- [Controls](/markdown/configuration/controls.md)
- [Coordinator](/markdown/configuration/coordinators.md)
- [Embeddings](/markdown/configuration/embeddings.md)
- [LLM Providers](/markdown/configuration/llm_providers.md)
- [Prompts](/markdown/configuration/prompts.md)
- [Reports](/markdown/configuration/reports.md)
- [Data Sources](/markdown/configuration/sources.md)
- [Tools](/markdown/configuration/tools.md)
- [UI](/markdown/configuration/ui.md)
- [Vector Stores](/markdown/configuration/vector_stores.md)

## YAML Spec

Detailed reference for writing a Lumen dashboard spec by hand.

- [Authentication](/markdown/configuration/spec/authentication.md)
- [Core Concepts](/markdown/configuration/spec/concepts.md)
- [Custom Components](/markdown/configuration/spec/customization.md)
- [Deployment](/markdown/configuration/spec/deployment.md)
- [Data Downloads](/markdown/configuration/spec/downloads.md)
- [Overview](/markdown/configuration/spec/index.md)
- [Transforming Data](/markdown/configuration/spec/pipelines.md)
- [Python API](/markdown/configuration/spec/python-api.md)
- [Loading Data](/markdown/configuration/spec/sources.md)
- [Variables & References](/markdown/configuration/spec/variables.md)
- [Visualizing Data](/markdown/configuration/spec/views.md)

## API Reference

Python API reference for Lumen's pipeline, sources, transforms, views, and AI components.

- [Agents](/markdown/reference/api/ai/agents.md)
- [Coordinator](/markdown/reference/api/ai/coordinator.md)
- [Core](/markdown/reference/api/ai/core.md)
- [Models](/markdown/reference/api/ai/models.md)
- [Tools](/markdown/reference/api/ai/tools.md)
- [Pipeline](/markdown/reference/api/pipeline.md)
- [Sources](/markdown/reference/api/sources.md)
- [Transforms](/markdown/reference/api/transforms.md)
- [Views](/markdown/reference/api/views.md)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants