Skip to content

feat(typing): Introduce PluginName NewType so plugin backends can type check #3794

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FBruzzesi merged 4 commits into from
Jul 17, 2026
Merged

feat(typing): Introduce PluginName NewType so plugin backends can type check #3794

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cc139e7
feat(typing): introduce `PluginName` NewType so plugin backends can t…
FBruzzesi Jul 14, 2026
d142211
keeps using generic
FBruzzesi Jul 15, 2026
a16cbb2
merge main
FBruzzesi Jul 16, 2026
782ca87
Feedback adjustments
FBruzzesi Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions src/narwhals/_typing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,18 @@
"""Type aliases describing the backends Narwhals dispatches to.

Built-in backends are enumerated as `Literal` unions (`Backend`, `EagerAllowed`, `LazyAllowed`, ...).
Plugin backends are discovered at runtime and cannot join those unions, so `PluginName` uses a
[`NewType`](https://typing.python.org/en/latest/spec/aliases.html#newtype) instead:
type checkers treat it as a distinct subtype of `str`, therefore:

- an arbitrary `str` is still rejected where a `backend` is expected, and
- a value explicitly wrapped as `PluginName("...")` is accepted.
"""

from __future__ import annotations

from types import ModuleType
from typing import TYPE_CHECKING, Literal
from typing import TYPE_CHECKING, Literal, NewType

from narwhals._typing_compat import TypeVar
from narwhals._utils import Implementation, _NoDefault
Expand Down Expand Up @@ -91,7 +102,17 @@
- An Implementation, such as: `Implementation.DASK`, `Implementation.PYSPARK`, ...
"""

BackendT = TypeVar("BackendT", bound=Backend)
PluginName = NewType("PluginName", str)
"""Name of a plugin's [entry point](https://packaging.python.org/en/latest/specifications/entry-points/).

- Wrap an entry point name to pass it wherever a `backend` is expected, e.g. `PluginName("my-plugin")`.
- Add it to a signature's `IntoBackend` parameter to advertise plugin support, e.g. `IntoBackend[EagerAllowed | PluginName]`.

Comment thread
dangotbanned marked this conversation as resolved.
See the `narwhals.plugins` module for how plugin authors register entry points

@FBruzzesi FBruzzesi Jul 16, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So... when I went to add the reference via [...][], I realized that we have no api-reference for plugins 😭 that's a needed follow-up

@dangotbanned dangotbanned Jul 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's a needed follow-up

Agreed!

For now, something I discovered with (microsoft/pylance-release#6611) - is that it supports lots of stuff (by default) that we either:

  • haven't even got configured for mkdocstrings
  • aren't actually possible
Cross ref to a function in the same module, but isn't API reference

mkdocstrings has a support for 2 kinds of cross-refs that we haven't got enabled.
IIRC, the syntax was different and it didn't work (yet?) in zensical.
But it still wouldn't be able to do this wizardry 😄

image

Cross ref to 3rd-party symbols

We can do something similar by adding inventories (if they have one ibis-project/ibis#11723)

But this let's you link links direct to the source code 🤯

image

Try these out

diff --git a/src/narwhals/_typing.py b/src/narwhals/_typing.py
index 189a1543b..c47190a26 100644
--- a/src/narwhals/_typing.py
+++ b/src/narwhals/_typing.py
@@ -110,6 +110,16 @@ PluginName = NewType("PluginName", str)
 
 See the `narwhals.plugins` module for how plugin authors register entry points
 and the contract a wrapped name must satisfy.
+
+
+## Some examples of working cross-refs
+Try these out in your IDE
+
+- [`importlib.metadata.EntryPoint.name`][]
+- [`IntoBackend`][]
+- [`EagerAllowed`][]
+- [`narwhals.typing.LazyAllowed`][]
+- [`narwhals.plugins.Plugin`][]
 """
 
 BackendT = TypeVar("BackendT", bound=Backend | PluginName)
image

@FBruzzesi FBruzzesi Jul 17, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't seem to work with modules though. Will merge without it for now and follow up with proper docs

and the contract a wrapped name must satisfy.
"""

BackendT = TypeVar("BackendT", bound=Backend | PluginName)

@dangotbanned dangotbanned Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this!

Much better idea than I had, while getting to the same goal 🥳

Edit: woops that was meant to start the review

IntoBackend: TypeAlias = BackendT | ModuleType
"""Anything that can be converted into a [`narwhals.Implementation`][].

Expand Down
5 changes: 2 additions & 3 deletions src/narwhals/_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@
from narwhals._typing import (
Backend,
IntoBackend,
PluginName,
_ArrowImpl,
_CuDFImpl,
_DaskImpl,
Expand Down Expand Up @@ -140,8 +141,6 @@
_SliceNone,
)

UnknownBackendName: TypeAlias = str
Comment thread
FBruzzesi marked this conversation as resolved.

FrameOrSeriesT = TypeVar(
"FrameOrSeriesT", bound=LazyFrame[Any] | DataFrame[Any] | Series[Any]
)
Expand Down Expand Up @@ -395,7 +394,7 @@ def from_string(cls: type[Self], backend_name: str) -> Implementation:

@classmethod
def from_backend(
cls: type[Self], backend: IntoBackend[Backend] | UnknownBackendName
cls: type[Self], backend: IntoBackend[Backend | PluginName]
) -> Implementation:
"""Instantiate from native namespace module, string, or Implementation.

Expand Down
19 changes: 14 additions & 5 deletions src/narwhals/dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,13 @@
from narwhals._compliant.typing import CompliantExprAny
from narwhals._expression_parsing import ExprMetadata
from narwhals._translate import IntoArrowTable
from narwhals._typing import EagerAllowed, IntoBackend, LazyAllowed, Polars
from narwhals._typing import (
EagerAllowed,
IntoBackend,
LazyAllowed,
PluginName,
Polars,
)
from narwhals.group_by import GroupBy, LazyGroupBy
from narwhals.typing import (
AsofJoinStrategy,
Expand Down Expand Up @@ -502,7 +508,10 @@ def __init__(self, df: Any, *, level: Literal["full", "lazy", "interchange"]) ->

@classmethod
def from_arrow(
cls, native_frame: IntoArrowTable, *, backend: IntoBackend[EagerAllowed]
cls,
native_frame: IntoArrowTable,
*,
backend: IntoBackend[EagerAllowed | PluginName],
) -> DataFrame[Any]:
"""Construct a DataFrame from an object which supports the PyCapsule Interface.

Expand Down Expand Up @@ -559,7 +568,7 @@ def from_dict(
data: Mapping[str, Any],
schema: IntoSchema | Mapping[str, IntoDType | None] | None = None,
*,
backend: IntoBackend[EagerAllowed] | None = None,
backend: IntoBackend[EagerAllowed | PluginName] | None = None,
) -> DataFrame[Any]:
"""Instantiate DataFrame from dictionary.

Expand Down Expand Up @@ -622,7 +631,7 @@ def from_dicts(
data: Sequence[Mapping[str, Any]],
schema: IntoSchema | Mapping[str, IntoDType | None] | None = None,
*,
backend: IntoBackend[EagerAllowed],
backend: IntoBackend[EagerAllowed | PluginName],
) -> DataFrame[Any]:
"""Instantiate DataFrame from a sequence of dictionaries representing rows.

Expand Down Expand Up @@ -696,7 +705,7 @@ def from_numpy(
data: _2DArray,
schema: IntoSchema | Sequence[str] | None = None,
*,
backend: IntoBackend[EagerAllowed],
backend: IntoBackend[EagerAllowed | PluginName],
) -> DataFrame[Any]:
"""Construct a DataFrame from a NumPy ndarray.

Expand Down
22 changes: 11 additions & 11 deletions src/narwhals/functions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@

from narwhals._native import NativeDataFrame, NativeLazyFrame, NativeSeries
from narwhals._translate import IntoArrowTable
from narwhals._typing import Backend, EagerAllowed, IntoBackend
from narwhals._typing import Backend, EagerAllowed, IntoBackend, PluginName
from narwhals.dataframe import DataFrame, LazyFrame
from narwhals.series import Series
from narwhals.typing import (
Expand Down Expand Up @@ -169,7 +169,7 @@ def new_series(
values: Any,
dtype: IntoDType | None = None,
*,
backend: IntoBackend[EagerAllowed],
backend: IntoBackend[EagerAllowed | PluginName],
) -> Series[Any]:
"""Instantiate Narwhals Series from iterable (e.g. list or array).

Expand Down Expand Up @@ -211,7 +211,7 @@ def _new_series_impl(
values: Any,
dtype: IntoDType | None = None,
*,
backend: IntoBackend[EagerAllowed],
backend: IntoBackend[EagerAllowed | PluginName],
) -> Series[Any]:
implementation = Implementation.from_backend(backend)
if is_eager_allowed(implementation):
Expand Down Expand Up @@ -241,7 +241,7 @@ def from_dict(
data: Mapping[str, Any],
schema: IntoSchema | Mapping[str, IntoDType | None] | None = None,
*,
backend: IntoBackend[EagerAllowed] | None = None,
backend: IntoBackend[EagerAllowed | PluginName] | None = None,
native_namespace: ModuleType | None = None, # noqa: ARG001
) -> DataFrame[Any]:
"""Instantiate DataFrame from dictionary.
Expand Down Expand Up @@ -332,7 +332,7 @@ def from_dicts(
data: Sequence[Mapping[str, Any]],
schema: IntoSchema | Mapping[str, IntoDType | None] | None = None,
*,
backend: IntoBackend[EagerAllowed],
backend: IntoBackend[EagerAllowed | PluginName],
) -> DataFrame[Any]:
"""Instantiate DataFrame from a sequence of dictionaries representing rows.

Expand Down Expand Up @@ -393,7 +393,7 @@ def from_numpy(
data: _2DArray,
schema: IntoSchema | Sequence[str] | None = None,
*,
backend: IntoBackend[EagerAllowed],
backend: IntoBackend[EagerAllowed | PluginName],
) -> DataFrame[Any]:
"""Construct a DataFrame from a NumPy ndarray.

Expand Down Expand Up @@ -482,7 +482,7 @@ def _is_into_schema(obj: Any) -> TypeIs[_IntoSchema]:


def from_arrow(
native_frame: IntoArrowTable, *, backend: IntoBackend[EagerAllowed]
native_frame: IntoArrowTable, *, backend: IntoBackend[EagerAllowed | PluginName]
) -> DataFrame[Any]: # pragma: no cover
"""Construct a DataFrame from an object which supports the PyCapsule Interface.

Expand Down Expand Up @@ -652,7 +652,7 @@ def _validate_separator_pyarrow(separator: str, **kwargs: Any) -> Any:
def read_csv(
source: FileSource,
*,
backend: IntoBackend[EagerAllowed],
backend: IntoBackend[EagerAllowed | PluginName],
separator: str = ",",
**kwargs: Any,
) -> DataFrame[Any]:
Expand Down Expand Up @@ -727,7 +727,7 @@ def read_csv(
def scan_csv(
source: FileSource,
*,
backend: IntoBackend[Backend],
backend: IntoBackend[Backend | PluginName],
separator: str = ",",
**kwargs: Any,
) -> LazyFrame[Any]:
Expand Down Expand Up @@ -813,7 +813,7 @@ def scan_csv(


def read_parquet(
source: FileSource, *, backend: IntoBackend[EagerAllowed], **kwargs: Any
source: FileSource, *, backend: IntoBackend[EagerAllowed | PluginName], **kwargs: Any
) -> DataFrame[Any]:
"""Read into a DataFrame from a parquet file.

Expand Down Expand Up @@ -886,7 +886,7 @@ def read_parquet(


def scan_parquet(
source: FileSource, *, backend: IntoBackend[Backend], **kwargs: Any
source: FileSource, *, backend: IntoBackend[Backend | PluginName], **kwargs: Any
) -> LazyFrame[Any]:
"""Lazily read from a parquet file.

Expand Down
16 changes: 15 additions & 1 deletion src/narwhals/plugins.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,24 @@
"""Runtime discovery of, and dispatch to, Narwhals plugin backends.

A plugin backend registers an [entry point](https://packaging.python.org/en/latest/specifications/entry-points/)
in the `narwhals.plugins` group. Plugins are discovered at runtime, so their names
cannot be enumerated in the `Literal` unions that describe built-in backends.

`PluginName` bridges that gap: a plugin's entry point name, wrapped as
`PluginName("my-plugin")`, is accepted wherever a `backend` is expected.

The contract for plugin authors is that the wrapped string **must** name an
installed plugin's entry point in the `narwhals.plugins` group.
"""

from __future__ import annotations

import sys
from functools import cache
from typing import TYPE_CHECKING, Any, Protocol

from narwhals._compliant import CompliantNamespace
from narwhals._typing import PluginName
from narwhals._typing_compat import TypeVar

if TYPE_CHECKING:
Expand All @@ -23,7 +37,7 @@
from narwhals.utils import Version


__all__ = ["Plugin", "from_native"]
__all__ = ["Plugin", "PluginName", "from_native"]

CompliantAny: TypeAlias = (
"CompliantDataFrameAny | CompliantLazyFrameAny | CompliantSeriesAny"
Expand Down
6 changes: 3 additions & 3 deletions src/narwhals/series.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
from typing_extensions import Self

from narwhals._compliant import CompliantSeries
from narwhals._typing import EagerAllowed, IntoBackend, NoDefault
from narwhals._typing import EagerAllowed, IntoBackend, NoDefault, PluginName
from narwhals.dataframe import DataFrame, MultiIndexSelector
from narwhals.dtypes import DType
from narwhals.typing import (
Expand Down Expand Up @@ -121,7 +121,7 @@ def from_numpy(
values: _1DArray,
dtype: IntoDType | None = None,
*,
backend: IntoBackend[EagerAllowed],
backend: IntoBackend[EagerAllowed | PluginName],
) -> Series[Any]:
"""Construct a Series from a NumPy ndarray.

Expand Down Expand Up @@ -186,7 +186,7 @@ def from_iterable(
values: Iterable[Any],
dtype: IntoDType | None = None,
*,
backend: IntoBackend[EagerAllowed],
backend: IntoBackend[EagerAllowed | PluginName],

@dangotbanned dangotbanned Jul 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Not a review, just trying to add context from our conversation on discord

If you are downstream from Narwhals and make use of our typing - we have been suggesting (#3149 (comment)) you write things like:

backend: IntoBackend[EagerAllowed]

Following this PR, that type describes exactly the same thing.
It does not indicate plugin support, meaning that there should be no expectation that downstream tests against plugins.

If downstream wants to opt-in to plugins, the update to their typing small:

- backend: IntoBackend[EagerAllowed]
+ backend: IntoBackend[EagerAllowed | PluginName]

But it gives a very clear signal (to their users) that plugins should work.
I hope that downstream (authors) will be more likely to update their tests, to account for plugins, when these concepts are visible in each signature with a backend 🙂

@FBruzzesi FBruzzesi Jul 16, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All this should go somewhere! but where exactly? 🧐

@FBruzzesi FBruzzesi Jul 16, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also made me realize that one should import from two modules:

from narwhals.typing import IntoBackend, EagerAllowed
from narwhals.plugins import PluginName

...

@dangotbanned dangotbanned Jul 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All this should go somewhere! but where exactly? 🧐

We could do either the PR description or the release notes until we find somewhere else?
For release notes I mean if we had a short highlight message in this style (https://github.com/vega/altair/releases#release-v5.5.0)

Both can work too 😅

) -> Series[Any]:
"""Construct a Series from an iterable.

Expand Down
Loading
Loading