Fix/issue 1895 by Vamshi1404 · Pull Request #2052 · holoviz/lumen

Vamshi1404 · 2026-08-21T17:05:57Z

Description

Clean up requirements in lumen by moving 17 dependencies from core to
optional extras, reducing the base install footprint from 29 to 12 core
dependencies.

Dependency classification

Dependency	Moved to	Evidence
`chardet`	`tests` extra	try/except fallback at `lumen/util.py:644` degrades to `'latin-1'`; used by `lumen/tests/transforms/test_sql.py:137` via `pytest.importorskip`
`duckdb`	new `duckdb` extra	only imported in `lumen/sources/duckdb.py:9` and `lumen/ai/vector_store.py:15`
`griffe`	new `ai` extra	only imported in `lumen/ai/translate.py:17`
`instructor`	new `ai` extra	only imported in `lumen/ai/{config,llm,utils}.py`
`intake`	new `intake` extra	only imported in `lumen/sources/intake.py:5`
`lxml`	removed	zero imports in `lumen/` source; already in `tests` extra
`markitdown`	new `ai` extra	only imported in `lumen/ai/{config,vector_store}.py` inside try/except
`nbformat`	new `ai` extra	only imported in `lumen/ai/export.py:9`
`panel-graphic-walker`	new `graphic-walker` extra	not imported at module load time; only string ref `'graphic_walker'` in `lumen/views/base.py:1758`
`panel-splitjs`	new `ai` extra	only imported in `lumen/ai/ui.py:32`
`pyarrow`	removed	zero direct imports; pandas uses it transparently as a parquet/CSV backend
`pydantic`	new `ai` extra	only imported in `lumen/ai/`
`pydantic-extra-types`	new `ai` extra	only imported in `lumen/ai/translate.py:24`
`semchunk`	new `ai` extra	only imported in `lumen/ai/vector_store.py`
`tabulate`	new `ai` extra	transitive runtime dependency of pandas/instructor in AI subsystem
`tiktoken`	new `ai` extra	only imported in `lumen/ai/utils.py:1335` inside try/except
`vl-convert-python`	new `ai` extra	only imported in `lumen/ai/editors.py:399` inside function

Remaining core (12)

bokeh, holoviews, hvplot, jinja2, narwhals, numpy, packaging, pandas, panel, panel-material-ui, param, sqlglot[c]

New optional extras

ai: instructor, griffe, markitdown, nbformat, pydantic, pydantic-extra-types, semchunk, tabulate, tiktoken, vl-convert-python, panel-splitjs
duckdb: duckdb
graphic-walker: panel-graphic-walker
intake: intake

pixi.toml changes

Root [dependencies] removed intake, lxml
[feature.ai.dependencies] removed chardet, duckdb, pyarrow, panel-graphic-walker; added tabulate
[feature.sql.dependencies] changed duckdb from * to >=1.2.0, added intake <2
New [feature.graphic-walker.dependencies] section for panel-graphic-walker >=0.6.4
Environments test-312/test-313 added graphic-walker feature
[feature.test-core.dependencies] added pyarrow, dask-core, intake <2 (these were transitively available before via core deps)

pyarrow — removed from core

A grep of lumen/ source shows zero direct import pyarrow statements — it is only referenced as a string in DATAFRAME_BACKENDS (lumen/util.py:110) and in comments. Moving it out is safe: users who need pyarrow can install it explicitly or via the tests extra.

chardet — moved to `tests` extra

chardet is imported inside a try/except ImportError at lumen/util.py:644-655 with a graceful 'latin-1' fallback, so the base install works without it. However, lumen/tests/transforms/test_sql.py:137 does pytest.importorskip("chardet"), so it belongs in the tests extra rather than being fully removed.

Docs status

Docs require follow-up but are out of scope (toml-only change). A grep of all .md files found:

docs/installation.md and docs/quick_start.md use provider-only extras (lumen[ai-openai], lumen[ai-anthropic], etc.) that do not chain in the base ai extra — pre-existing issue (provider extras never included lumen[ai] before this PR). Should be updated to lumen[ai,ai-openai] etc.
docs/configuration/spec/deployment.md:204-206 says pip install intake as a standalone command. Should reference pip install lumen[intake].
docs/contributing.md:70 says pip install -e ".[tests]" — no longer pulls in AI/sql/graphic-walker deps. Should be pip install -e ".[dev]".
Several tutorials reference the old standalone lumen-ai package name.
Multiple docs reference DuckDBSource without install instructions for the new duckdb extra.

None of these were changed in this PR but should be addressed in a follow-up.

Fixes #1895

AI Disclosure

Tool & Model: opencode + big-pickle
Usage: Dependency classification via grep analysis of lumen/ source tree; test execution and verification; PR description drafting. All classifications were verified by manual code inspection and confirmed by running the test suite locally.

I have tested all AI-generated content in my PR.
I take responsibility for all AI-generated content in my PR.

Checklist

Tests added and are passing
Added documentation edit this accordingly

codecov · 2026-08-21T18:24:06Z

Codecov Report

❌ Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.52%. Comparing base (2470a24) to head (4f58f5a).

Files with missing lines	Patch %	Lines
lumen/tests/ai/test_ui.py	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2052      +/-   ##
==========================================
- Coverage   75.53%   75.52%   -0.01%     
==========================================
  Files         214      214              
  Lines       39682    39687       +5     
==========================================
+ Hits        29974    29975       +1     
- Misses       9708     9712       +4

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…tras Move 17 dependencies from core to optional extras in pyproject.toml, reducing the base install footprint from 29 to 12 core dependencies. - New extras: ai, duckdb, graphic-walker, intake - Removed from core: chardet, duckdb, griffe, instructor, intake, lxml, markitdown, nbformat, panel-graphic-walker, panel-splitjs, pydantic, pydantic-extra-types, semchunk, tabulate, tiktoken, vl-convert-python - pyarrow removed from core (zero direct imports) - chardet moved to tests extra - pixi.toml updated: test-core feature includes intake, dask-core, pyarrow to compensate for removed core deps; new graphic-walker feature added Fixes holoviz#1895

Vamshi1404 · 2026-08-21T19:09:47Z

@ahuang11 Version constraints like intake <2 are pre-existing pins carried over from core. Adjusting or loosening version bounds is out of scope for this cleanup and should be tracked in a separate issue.

ahuang11 · 2026-08-21T19:16:40Z

Can we drop intake?

Vamshi1404 · 2026-08-21T19:26:04Z

@ahuang11 intake — IntakeSource, IntakeSQLSource, and IntakeDremioSQLSource are kept for backward compatibility, but lumen's built-in sources (DuckDBSource, SQLAlchemySource, FileSource) cover the same use cases without an extra dependency. The intake sources are effectively redundant at this point. Removing them would be a breaking change for anyone with type: intake in their specs, so i suggest it should be tracked as a separate issue with a deprecation cycle rather than bundled into this cleanup.

ahuang11 · 2026-08-21T19:37:37Z

Make it an optional dep*

Vamshi1404 · 2026-08-22T02:57:16Z

Yes, already done. intake is now an optional extra

ahuang11 · 2026-08-24T21:36:14Z

I think panel-graphic-walker is essential for the Data source tab

ahuang11 · 2026-08-25T16:03:14Z

duckdb is also core to using lumen ai

…pendencies in core

Vamshi1404 · 2026-08-25T18:18:04Z

Good catch @ahuang11 , thanks! I went back through all the deps and moved duckdb, panel-graphic-walker, panel-splitjs, chardet, and pyarrow back into core so we don't break default UI tabs, vector stores, or file loading.

The rest of the optional stuff (LLM providers, intake, lxml) is kept in extras. Pushed the update!

ahuang11 · 2026-08-25T18:19:00Z

Actually, I think AI should NOT be separate dependencies; Lumen has pivoted towards full AI.


ai = [
    'instructor >=1.14.3',
    'griffe',
    'markitdown',
    'nbformat',
    'pydantic >=2.8.0',
    'pydantic-extra-types',
    'semchunk',
    'tabulate',
    'tiktoken',
    'vl-convert-python',
]

…ly intake and lxml to extras

…ction

…ithout modifying test files

chardet is imported inside try/except ImportError in lumen/util.py with a graceful latin-1 fallback - the base install works without it. The only test exercising it already uses pytest.importorskip('chardet'). Belongs in the tests extra, not core dependencies.

Vamshi1404 force-pushed the fix/issue-1895 branch from 0cdb465 to 84222f6 Compare August 21, 2026 18:49

Vamshi1404 force-pushed the fix/issue-1895 branch from 3e03cc6 to f6a9d40 Compare August 21, 2026 18:55

Merge branch 'main' into fix/issue-1895

b1cb5f0

Merge branch 'main' into fix/issue-1895

e34de29

Merge branch 'main' into fix/issue-1895

baea876

Fix: Re-audit dependencies to retain core UI, DuckDB, and encoding de…

358c2ef

…pendencies in core

Merge branch 'main' into fix/issue-1895

bf99534

Vamshi1404 added 6 commits August 26, 2026 00:00

Fix: Keep AI dependencies in core per maintainer directive, moving on…

cd236e5

…ly intake and lxml to extras

Fix: Safely import SQLAlchemySource in test_ui.py for test-core colle…

89131f9

…ction

style: format imports in test_ui.py for isort

3d2ccb0

test: skip sqlite resolve_data tests when sqlalchemy is not installed

eca45ca

Fix: Add sqlalchemy to test-core dependencies in pixi and pyproject w…

77e24d2

…ithout modifying test files

Navigation Menu

Uh oh!

Uh oh!

Fix/issue 1895 - #2052

Fix/issue 1895#2052
Vamshi1404 wants to merge 12 commits into
holoviz:mainfrom
Vamshi1404:fix/issue-1895

Vamshi1404 commented Aug 21, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 21, 2026 •

edited

Loading

Uh oh!

Vamshi1404 commented Aug 21, 2026

Uh oh!

ahuang11 commented Aug 21, 2026

Uh oh!

Vamshi1404 commented Aug 21, 2026 •

edited

Loading

Uh oh!

ahuang11 commented Aug 21, 2026

Uh oh!

Vamshi1404 commented Aug 22, 2026

Uh oh!

ahuang11 commented Aug 24, 2026

Uh oh!

ahuang11 commented Aug 25, 2026

Uh oh!

Vamshi1404 commented Aug 25, 2026

Uh oh!

ahuang11 commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

Vamshi1404 commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Dependency classification

Remaining core (12)

New optional extras

pixi.toml changes

pyarrow — removed from core

chardet — moved to tests extra

Docs status

AI Disclosure

Checklist

Uh oh!

codecov Bot commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Vamshi1404 commented Aug 21, 2026

Uh oh!

ahuang11 commented Aug 21, 2026

Uh oh!

Vamshi1404 commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahuang11 commented Aug 21, 2026

Uh oh!

Vamshi1404 commented Aug 22, 2026

Uh oh!

ahuang11 commented Aug 24, 2026

Uh oh!

ahuang11 commented Aug 25, 2026

Uh oh!

Vamshi1404 commented Aug 25, 2026

Uh oh!

ahuang11 commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Vamshi1404 commented Aug 21, 2026 •

edited

Loading

chardet — moved to `tests` extra

codecov Bot commented Aug 21, 2026 •

edited

Loading

Vamshi1404 commented Aug 21, 2026 •

edited

Loading