feat: Extend plugin support for IO and from_* methods/functions

docs/api-reference/exceptions.md

@@ -11,6 +11,7 @@
         - InvalidOperationError
         - MultiOutputExpressionError
         - NarwhalsUnstableWarning
+        - PluginError
         - ShapeError
         - UnsupportedDTypeError
       show_source: false

docs/api-reference/index.md

@@ -26,3 +26,4 @@
 - [narwhals.selectors](selectors.md)
 - [narwhals.typing](typing.md)
 - [narwhals.utils](utils.md)
+- [narwhals.plugins](plugins.md)

docs/api-reference/plugins.md

@@ -0,0 +1,13 @@
+# `narwhals.plugins`
+
+For an overview of how to write a plugin, see [extensions and plugins](../extending.md).
+
+::: narwhals.plugins
+    handler: python
+    options:
+      members:
+        - Plugin
+        - PluginName
+        - from_native
+      show_source: false
+      show_bases: false

docs/api-reference/typing.md

@@ -21,6 +21,8 @@ Narwhals comes fully statically typed. In addition to `nw.DataFrame`, `nw.Expr`,
         - Backend
         - EagerAllowed
         - LazyAllowed
+        - FileSource
+        - NormalizedPath
         - IntoDType
         - IntoSchema
         - SizeUnit

docs/extending.md

@@ -60,11 +60,52 @@ handle plugins. For this integration to work, any plugin architecture must conta
     Take a look at the `Plugin` protocol in `narwhals/plugins.py` for the
     signatures.
 
-## IO functions: the namespace contract
+## Supporting `backend=...` in Narwhals functions
+
+Functions and constructors which accept a `backend` argument can also dispatch to a
+plugin. Users can pass:
+
+- the plugin's entry point name (e.g. `backend="narwhals-grizzlies"`),
+- the plugin's module name (e.g. `backend="narwhals_grizzlies"`),
+- or the plugin's module itself (e.g. `backend=narwhals_grizzlies`).
+
+In all cases, dispatch goes through the compliant namespace returned by the plugin's
+`__narwhals_namespace__`:
+
+1. **IO functions** (`read_csv`, `scan_csv`, `read_parquet`, `scan_parquet`): these
+   call same-named methods on the compliant namespace, following the
+   [namespace contract](#io-functions-the-namespace-contract) below. If the plugin's
+   namespace does not implement the required method, an informative
+   [`PluginError`](api-reference/exceptions.md) is raised.
+
+2. **Eager constructors** (`from_dict`, `from_dicts`, `from_numpy`, `from_arrow`,
+   `new_series`, as well as the `DataFrame.from_*` and `Series.from_*` classmethods):
+   these are eager-only. If the plugin's compliant namespace implements the
+   `EagerNamespace` protocol (in particular the `_dataframe` and `_series` properties,
+   and the `from_dict`, `from_dicts`, `from_numpy`, `from_arrow` and `from_iterable`
+   constructors on the respective compliant classes), these functions work with no
+   extra plugin code. Lazy-only plugins get an informative
+   [`PluginError`](api-reference/exceptions.md) instead.
+
+Methods which internally construct Series (for example `Series.scatter`, or
+`DataFrame.filter` with a list of booleans) use the compliant namespace of the object
+they are called on, so they also work for eager plugins.
+
+!!! tip "Type checking"
+
+    The `backend` parameters of these functions are typed with
+    [`PluginName`](api-reference/plugins.md), a `str`
+    [`NewType`](https://docs.python.org/3/library/typing.html#newtype): plugin names
+    are only known at runtime, so an opaque string does not type check, but an
+    explicitly wrapped one does, e.g.
+    `nw.from_dict(data, backend=PluginName("narwhals-grizzlies"))`.
+    Passing the plain string works at runtime all the same.
+
+### IO functions: the namespace contract
 
 The Narwhals IO functions (`read_csv`, `scan_csv`, `read_parquet`, `scan_parquet`)
 dispatch to same-named methods on the compliant namespace. This is a single mechanism,
-shared by built-in backends and extensions alike: to support these functions, a
+shared by built-in backends and plugins alike: to support these functions, a
 compliant namespace implements (a subset of):
 
 ```py

packages/test-plugin/src/test_plugin/dataframe.py

@@ -11,15 +11,91 @@
 from narwhals.typing import CompliantLazyFrame
 
 if TYPE_CHECKING:
+    from collections.abc import Mapping, Sequence
     from typing import TypeAlias
 
     from typing_extensions import Self
 
-    from narwhals import LazyFrame  # noqa: F401
+    from narwhals import DataFrame, LazyFrame
+    from narwhals._utils import _LimitedContext
 
 DictFrame: TypeAlias = dict[str, list[Any]]
 
 
+class DictDataFrame:
+    """Minimal eager frame, kept to the smallest surface exercised in narwhals' tests."""
+
+    _implementation = Implementation.UNKNOWN
+
+    def __init__(self, native_dataframe: DictFrame, *, version: Version) -> None:
+        self._native_frame: DictFrame = native_dataframe
+        self._version = version
+
+    @classmethod
+    def from_dict(
+        cls,
+        data: Mapping[str, Any],
+        /,
+        *,
+        context: _LimitedContext,
+        schema: Any = None,  # noqa: ARG003
+    ) -> Self:
+        return cls(
+            {name: list(values) for name, values in data.items()},
+            version=context._version,
+        )
+
+    @classmethod
+    def from_dicts(
+        cls,
+        data: Sequence[Mapping[str, Any]],
+        /,
+        *,
+        context: _LimitedContext,
+        schema: Any = None,  # noqa: ARG003
+    ) -> Self:
+        columns: list[str] = list(data[0]) if data else []
+        return cls(
+            {name: [row[name] for row in data] for name in columns},
+            version=context._version,
+        )
+
+    @classmethod
+    def from_numpy(
+        cls, data: Any, /, *, context: _LimitedContext, schema: Any = None
+    ) -> Self:
+        names = (
+            list(schema)
+            if schema is not None
+            else [str(index) for index in range(data.shape[1])]
+        )
+        return cls(
+            {name: data[:, index].tolist() for index, name in enumerate(names)},
+            version=context._version,
+        )
+
+    @classmethod
+    def from_arrow(cls, data: Any, /, *, context: _LimitedContext) -> Self:
+        import pyarrow as pa
+
+        return cls(pa.table(data).to_pydict(), version=context._version)
+
+    def __narwhals_dataframe__(self) -> Self:
+        return self
+
+    def __narwhals_namespace__(self) -> Any:
+        from test_plugin.namespace import DictNamespace
+
+        return DictNamespace(version=self._version)
+
+    @property
+    def native(self) -> DictFrame:
+        return self._native_frame
+
+    def to_narwhals(self) -> DataFrame[Any]:
+        return self._version.dataframe(self, level="full")
+
+
 class DictLazyFrame(
     CompliantLazyFrame[Any, "DictFrame", "LazyFrame[DictFrame]"],  # type: ignore[type-var]
     ValidateBackendVersion,
@@ -33,6 +109,9 @@ def __init__(self, native_dataframe: DictFrame, *, version: Version) -> None:
     def __narwhals_lazyframe__(self) -> Self:
         return self
 
+    def to_narwhals(self) -> LazyFrame[Any]:
+        return self._version.lazyframe(self, level="lazy")
+
     @property
     def columns(self) -> list[str]:  # pragma: no cover
         return list(self._native_frame.keys())
@@ -74,7 +153,6 @@ def _with_version(self, version: Version) -> Self:
     sink_parquet = not_implemented()
     sort = not_implemented()
     tail = not_implemented()
-    to_narwhals = not_implemented()
     unique = not_implemented()
     unpivot = not_implemented()
     with_columns = not_implemented()

packages/test-plugin/src/test_plugin/namespace.py

@@ -3,11 +3,13 @@
 from typing import TYPE_CHECKING, Any
 
 from narwhals._compliant import CompliantNamespace
-from narwhals._utils import not_implemented
-from test_plugin.dataframe import DictFrame, DictLazyFrame
+from narwhals._utils import Implementation, not_implemented
+from test_plugin.dataframe import DictDataFrame, DictFrame, DictLazyFrame
 
 if TYPE_CHECKING:
+    from narwhals.typing import NormalizedPath
     from narwhals.utils import Version
+    from test_plugin.series import DictSeries
 
 
 class DictNamespace(CompliantNamespace[DictLazyFrame, Any]):
@@ -17,9 +19,44 @@ def __init__(self, *, version: Version) -> None:
     def from_native(self, native_object: DictFrame) -> DictLazyFrame:
         return DictLazyFrame(native_object, version=self._version)
 
+    @property
+    def _dataframe(self) -> type[DictDataFrame]:
+        return DictDataFrame
+
+    @property
+    def _series(self) -> type[DictSeries]:
+        from test_plugin.series import DictSeries
+
+        return DictSeries
+
+    # IO methods below follow the namespace contract used by `narwhals.functions`
+    # (see "IO functions: the namespace contract" in `docs/extending.md`).
+    # `read_*` are deliberately left unimplemented: `test_plugin` wraps dicts
+    # lazily, so it only supports `scan_*`.
+
+    def scan_csv(
+        self, source: NormalizedPath, *, separator: str = ",", **kwds: Any
+    ) -> DictLazyFrame:
+        import csv
+        from pathlib import Path
+
+        with Path(source).open(newline="", encoding="utf-8") as file:
+            header, *rows = list(csv.reader(file, delimiter=separator))
+        data = {name: [row[index] for row in rows] for index, name in enumerate(header)}
+        return DictLazyFrame(data, version=self._version)
+
+    def scan_parquet(self, source: NormalizedPath, **kwds: Any) -> DictLazyFrame:
+        import pyarrow.parquet as pq
+
+        data: DictFrame = pq.read_table(source, **kwds).to_pydict()
+        return DictLazyFrame(data, version=self._version)
+
+    # NOTE: `not_implemented.__get__` reads `instance._implementation` to build its
+    # error message, so `_implementation` itself must be a real value.
+    _implementation = Implementation.UNKNOWN
+
     is_native: Any = not_implemented()
     _expr: Any = not_implemented()
-    _implementation: Any = not_implemented()
     corr: Any = not_implemented()
     cov: Any = not_implemented()
     len: Any = not_implemented()
@@ -37,5 +74,3 @@ def from_native(self, native_object: DictFrame) -> DictLazyFrame:
     selectors: Any = not_implemented()
     coalesce: Any = not_implemented()
     struct: Any = not_implemented()
-    scan_csv: Any = not_implemented()
-    scan_parquet: Any = not_implemented()

packages/test-plugin/src/test_plugin/series.py

@@ -0,0 +1,73 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+from narwhals._utils import Implementation
+
+if TYPE_CHECKING:
+    from collections.abc import Iterable
+
+    from typing_extensions import Self
+
+    from narwhals._utils import Version, _LimitedContext
+    from narwhals.series import Series
+
+
+class DictSeries:
+    """Minimal eager series, kept to the smallest surface exercised in narwhals' tests."""
+
+    _implementation = Implementation.UNKNOWN
+
+    def __init__(
+        self, values: Iterable[Any], *, name: str = "", version: Version
+    ) -> None:
+        self._values: list[Any] = list(values)
+        self._name = name
+        self._version = version
+
+    @classmethod
+    def from_iterable(
+        cls,
+        data: Iterable[Any],
+        /,
+        *,
+        context: _LimitedContext,
+        name: str = "",
+        dtype: Any = None,  # noqa: ARG003
+    ) -> Self:
+        return cls(data, name=name, version=context._version)
+
+    @classmethod
+    def from_numpy(cls, data: Any, /, *, context: _LimitedContext) -> Self:
+        return cls(data.tolist(), version=context._version)
+
+    def __narwhals_series__(self) -> Self:
+        return self
+
+    def __narwhals_namespace__(self) -> Any:
+        from test_plugin.namespace import DictNamespace
+
+        return DictNamespace(version=self._version)
+
+    @property
+    def native(self) -> list[Any]:
+        return self._values
+
+    @property
+    def name(self) -> str:
+        return self._name
+
+    def alias(self, name: str) -> Self:
+        return self.__class__(self._values, name=name, version=self._version)
+
+    def is_empty(self) -> bool:
+        return not self._values
+
+    def scatter(self, indices: Self, values: Self) -> Self:
+        data = list(self._values)
+        for index, value in zip(indices.native, values.native, strict=True):
+            data[index] = value
+        return self.__class__(data, name=self._name, version=self._version)
+
+    def to_narwhals(self) -> Series[Any]:
+        return self._version.series(self, level="full")