Remove non-array api dpnp support by ethanglaser · Pull Request #3322 · uxlfoundation/scikit-learn-intelex

ethanglaser · 2026-07-13T20:29:09Z

Description

Removes direct support for non-array API dpnp support. Migrates all examples and spmd tests to set array_api_dispatch=True for fully on-device handling without conversion to host. Using dpnp inputs without setting array_api_dispatch now returns numpy arrays.

Checklist:

Completeness and readability

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes or created a separate PR with updates and provided its number in the description, if necessary.
Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
I have resolved any merge conflicts that might occur with the base branch.

Testing

I have run it locally and tested the changes extensively.
All CI jobs are green or I have provided justification why they aren't.
I have extended testing suite if new functionality was introduced in this PR.

Performance

I have measured performance for affected algorithms using scikit-learn_bench and provided at least a summary table with measured data, if performance change is expected.
I have provided justification why performance and/or quality metrics have changed or why changes are not expected.
I have extended the benchmarking suite and provided a corresponding scikit-learn_bench PR if new measurable functionality was introduced in this PR.

ethanglaser · 2026-07-13T20:29:24Z

/intelci: run

ethanglaser · 2026-07-13T23:27:48Z

/intelci: run

ethanglaser · 2026-07-13T23:28:49Z

/intelci: run

codecov · 2026-07-13T23:56:53Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag	Coverage Δ
azure	`82.17% <100.00%> (+0.06%)`	⬆️
github	`74.33% <100.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
onedal/covariance/covariance.py	`88.23% <ø> (-0.34%)`	⬇️
onedal/datatypes/__init__.py	`100.00% <ø> (ø)`
onedal/decomposition/pca.py	`95.83% <ø> (-0.09%)`	⬇️
onedal/utils/_array_api.py	`95.83% <ø> (+8.02%)`	⬆️
sklearnex/_device_offload.py	`83.33% <100.00%> (+3.18%)`	⬆️
sklearnex/basic_statistics/basic_statistics.py	`88.67% <100.00%> (+0.21%)`	⬆️
...x/basic_statistics/incremental_basic_statistics.py	`98.83% <100.00%> (+0.01%)`	⬆️
sklearnex/cluster/dbscan.py	`91.80% <100.00%> (+0.13%)`	⬆️
sklearnex/cluster/k_means.py	`84.21% <100.00%> (+0.08%)`	⬆️
sklearnex/covariance/incremental_covariance.py	`81.54% <100.00%> (+1.54%)`	⬆️
... and 19 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ethanglaser · 2026-07-14T00:46:49Z

/intelci: run

ethanglaser · 2026-07-14T06:45:13Z

/intelci: run

david-cortes-intel · 2026-07-14T10:28:23Z

Please update also these doc pages:
https://uxlfoundation.github.io/scikit-learn-intelex/2026.1/index.html
https://uxlfoundation.github.io/scikit-learn-intelex/2026.1/array_api.html

icfaust · 2026-07-14T15:06:29Z

    return wrapper_impl
-
-
-def support_sycl_format(func):


Necessary change, but I wanted to highlight IntelPython/dpctl#2131 where after this change things are going to require the use of experimental scikit-learn features for use of dpnp/dpctl with Scikit-learn-intelex (also already the case in scikit-learn support).

Derived from this line of code (which was originally also in dpctl tensors)
https://github.com/IntelPython/dpnp/blob/75453b62b0d8add89ad32aafbc3ef409bb13b7bc/dpnp/dpnp_array.py#L174

@ndgrigorian

icfaust · 2026-07-14T15:12:15Z

                _check_set_output_transform(est, method, X, estimator)

    else:
+        # For dpnp, the method is only exercised under array_api_dispatch in the


I think standard estimator patching and special estimator patching should now become split into two separate tests, one with array api enabled (using a pytest.fixture available in the conftest.py for dpnp, torch, array_api_strict, etc.), and one without (for use of pandas, numpy).

I hope this isn't annoying, but its time.

Its been a while since I've looked at these tests that I refactored a long time ago, and its mentioning features that I didn't think was possible or intentional. I can give it a deeper look, but I recommend the split instead (where fit is under array api dispatch).

Check if current state aligns with what you had in mind

icfaust

I know its still draft, but I wanted to help steer things a bit. Hope its useful.

ethanglaser · 2026-07-14T15:31:48Z

I know its still draft, but I wanted to help steer things a bit. Hope its useful.

Much appreciated!

ethanglaser · 2026-07-14T17:44:06Z

/intelci: run

ethanglaser · 2026-07-14T19:28:48Z

/intelci: run

david-cortes-intel

@ethanglaser I believe you also need to remove this:

scikit-learn-intelex/sklearnex/_device_offload.py

Line 308 in cb209a7

def support_sycl_format(func):

It might also require modifying some code paths in the other functions from that file.

david-cortes-intel · 2026-07-15T06:43:17Z

+@pytest.mark.parametrize("dataframe, queue", get_dataframes_and_queues("dpnp,array_api"))
+@pytest.mark.parametrize("estimator, method", gen_models_info(PATCHED_MODELS))
+def test_standard_estimator_patching_array_api(
+    with_array_api, caplog, dataframe, queue, dtype, estimator, method


Since you're modifying these: array API tests should be skipped on numpy<2.2.

Updated. But I believe it's numpy 2.1.

The error messages typically show a branch with <=2.1. Maybe @yuejiaointel could comment here.

Numpy 2.1 has array api-related updates
https://numpy.org/devdocs/release/2.1.0-notes.html
I don't think numpy 2.2 has any
https://numpy.org/devdocs/release/2.2.0-notes.html

But the code branches in pytorch that show up in error messages have numpy<=2.1.

If you're able to share a link to CI fail / reproducer for something that fails with numpy 2.1 that would be great

You won't see those in CI logs, because those don't run torch. But you would see them if running things locally on torch + GPU and remove some of the current pytest skips.

The test currently doesn't run torch. Lets cross that bridge when we come to it (i.e. when we make torch more common in central sklearnex testing). I'd imagine that sklearn conformance testing would likely complain first.

ethanglaser · 2026-07-15T22:40:59Z

@ethanglaser I believe you also need to remove this:

scikit-learn-intelex/sklearnex/_device_offload.py

Line 308 in cb209a7

def support_sycl_format(func):

It might also require modifying some code paths in the other functions from that file.

It was already removed

ethanglaser · 2026-07-15T22:41:38Z

/intelci: run

david-cortes-intel · 2026-07-16T12:23:22Z

@ethanglaser I believe you also need to remove this:

scikit-learn-intelex/sklearnex/_device_offload.py

Line 308 in cb209a7

def support_sycl_format(func):

It might also require modifying some code paths in the other functions from that file.

It was already removed

What about the other functions? Are they still needed for target offload?

ethanglaser · 2026-07-16T18:31:05Z

What about the other functions? Are they still needed for target offload?

dispatch/_get_backend/wrap_output_data are still needed for target offload - that path was never the dpnp-copy path. support_input_format is still needed for its output namespace conversion, but its input-side device/queue path (the sycl_usm_array_interface block at L273-275 and the kwargs.get("queue") device branch) is dead now that it only wraps host-side sklearn/daal4py fallbacks - none of which take a queue, so I removed those - please check

Copilot

Pull request overview

This PR removes the project’s custom/non-standard dpnp “device array” compatibility layer in favor of relying on scikit-learn’s Array API dispatch (sklearn.utils._array_api.get_namespace + array_api_dispatch=True). It updates runtime code paths, tests, examples, and docs so dpnp is only supported as an Array API framework (and is otherwise treated as an unsupported device input that is host-transferred).

Changes:

Replace sklearnex.utils._array_api.get_namespace and oneDAL SYCL-namespace helpers with scikit-learn’s sklearn.utils._array_api.get_namespace.
Update tests/examples/docs to require config_context(array_api_dispatch=True) for dpnp/array-api inputs.
Improve SYCL USM table conversion correctness by applying __sycl_usm_array_interface__["offset"] during pointer conversion.

Reviewed changes

Copilot reviewed 71 out of 71 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
sklearnex/utils/validation.py	Switch `get_namespace` import to scikit-learn Array API utility.
sklearnex/utils/class_weight.py	Switch `get_namespace` import to scikit-learn Array API utility.
sklearnex/utils/_array_api.py	Remove custom namespace selection wrapper; rely on scikit-learn `get_namespace`.
sklearnex/tests/test_patching.py	Split array-api vs non-array-api patching tests; add tolerated sklearn array-api gaps list.
sklearnex/svm/_classes.py	Switch to scikit-learn `get_namespace`; keep sklearnex `enable_array_api`.
sklearnex/svm/_base.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/spmd/neighbors/tests/test_neighbors_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/neighbors/neighbors.py	Switch to scikit-learn `get_namespace`.
sklearnex/spmd/linear_model/tests/test_logistic_regression_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/linear_model/tests/test_linear_regression_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/linear_model/tests/test_incremental_linear_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/ensemble/tests/test_forest_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/decomposition/tests/test_pca_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/decomposition/tests/test_incremental_pca_spmd.py	Always enable `array_api_dispatch=True`; run deferred-finalize attribute access under dispatch.
sklearnex/spmd/covariance/tests/test_incremental_covariance_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/covariance/tests/test_covariance_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/cluster/tests/test_kmeans_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/cluster/tests/test_dbscan_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/basic_statistics/tests/test_incremental_basic_statistics_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/spmd/basic_statistics/tests/test_basic_statistics_spmd.py	Always enable `array_api_dispatch=True` (drop False branch).
sklearnex/preview/preprocessing/tests/test_data.py	Gate dpnp/array-api inputs behind `array_api_dispatch=True`; normalize fitted-attr comparisons via `_as_numpy`.
sklearnex/preview/preprocessing/_data.py	Remove `support_sycl_format` fallback wrapper; switch to scikit-learn `get_namespace`.
sklearnex/preview/decomposition/incremental_pca.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/preview/covariance/covariance.py	Remove `support_sycl_format` wrappers; switch to scikit-learn `get_namespace`; simplify dpnp-without-dispatch handling.
sklearnex/neighbors/knn_unsupervised.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/neighbors/knn_regression.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/neighbors/knn_classification.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/neighbors/common.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/neighbors/_lof.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/linear_model/ridge.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/linear_model/logistic_regression.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/linear_model/linear.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/linear_model/incremental_ridge.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/linear_model/incremental_linear.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/linear_model/_base_linear_model.py	Switch to scikit-learn `get_namespace`.
sklearnex/ensemble/_forest.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/dummy/_dummy.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/decomposition/pca.py	Remove `support_sycl_format` usage; switch to scikit-learn `get_namespace`.
sklearnex/covariance/tests/test_incremental_covariance.py	Update test logic to use `array_api_dispatch=True` for dpnp/array-api inputs.
sklearnex/covariance/incremental_covariance.py	Remove `support_sycl_format` usage; switch to scikit-learn `get_namespace`.
sklearnex/cluster/k_means.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/cluster/dbscan.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/basic_statistics/incremental_basic_statistics.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/basic_statistics/basic_statistics.py	Switch to scikit-learn `get_namespace`; remove local import.
sklearnex/_device_offload.py	Remove dpnp-copy backpath and `support_sycl_format`; align offload/fallback behavior with Array API dispatch.
onedal/utils/_array_api.py	Remove SYCL namespace detection helpers (dpnp special casing).
onedal/decomposition/pca.py	Remove unused SYCL-namespace import.
onedal/datatypes/tests/test_data.py	Use `__array_namespace__()` directly instead of removed `_get_sycl_namespace`.
onedal/datatypes/sycl_usm/sycl_usm_utils.hpp	Add declaration for SUA `offset` accessor.
onedal/datatypes/sycl_usm/sycl_usm_utils.cpp	Implement SUA `offset` accessor (for sliced views).
onedal/datatypes/sycl_usm/data_conversion.cpp	Apply SUA `offset` when computing the data pointer for table conversion.
onedal/datatypes/_sycl_usm.py	Remove dpnp copy helper module.
onedal/datatypes/init.py	Remove `copy_to_dpnp` from public API exports.
onedal/covariance/covariance.py	Remove unused SYCL-namespace import.
examples/sklearnex/random_forest_regressor_spmd.py	Wrap dpnp GPU runs in `config_context(array_api_dispatch=True)`.
examples/sklearnex/random_forest_classifier_spmd.py	Wrap dpnp GPU runs in `config_context(array_api_dispatch=True)`.
examples/sklearnex/pca_spmd.py	Wrap dpnp GPU runs in `config_context(array_api_dispatch=True)`.
examples/sklearnex/logistic_regression_spmd.py	Wrap dpnp GPU runs in `config_context(array_api_dispatch=True)`.
examples/sklearnex/linear_regression_spmd.py	Wrap dpnp GPU runs in `config_context(array_api_dispatch=True)`.
examples/sklearnex/knn_bf_regression_spmd.py	Wrap dpnp GPU runs in `config_context(array_api_dispatch=True)`.
examples/sklearnex/knn_bf_classification_spmd.py	Wrap dpnp GPU runs in `config_context(array_api_dispatch=True)`.
examples/sklearnex/kmeans_spmd.py	Wrap dpnp GPU runs in `config_context(array_api_dispatch=True)`.
examples/sklearnex/incremental_pca_dpnp.py	Wrap dpnp GPU usage in `config_context(array_api_dispatch=True)`.
examples/sklearnex/incremental_linear_regression_dpnp.py	Wrap dpnp GPU usage in `config_context(array_api_dispatch=True)`.
examples/sklearnex/incremental_covariance_spmd.py	Wrap dpnp GPU usage in `config_context(array_api_dispatch=True)`.
examples/sklearnex/incremental_basic_statistics_dpnp.py	Wrap dpnp GPU usage in `config_context(array_api_dispatch=True)`.
examples/sklearnex/dbscan_spmd.py	Wrap dpnp GPU usage in `config_context(array_api_dispatch=True)`.
examples/sklearnex/covariance_spmd.py	Wrap dpnp GPU usage in `config_context(array_api_dispatch=True)`.
examples/sklearnex/basic_statistics_spmd.py	Wrap dpnp GPU usage in `config_context(array_api_dispatch=True)`.
doc/sources/oneapi-gpu.rst	Document dpnp as Array API-only; clarify host fallback when dispatch is disabled.
doc/sources/input-types.rst	Document dpnp support only when array API mode is enabled; adjust fallback description.

@@ -303,27 +286,3 @@ def wrapper_impl(*args, **kwargs):
        return result


icfaust · 2026-07-20T15:25:47Z

-                )
-                _check_output_type(result2, y2, method, estimator, caplog, X=X2, est=est)
-                _check_fitted_attributes(est, X2, estimator, caplog, queue=queue)
+    except Exception as e:


Not your fault, but this test is starting to get hard to follow. I refactored this code long ago, and now I don't even fully know what's going on... :( . Ideally it should run the code once as a fixture, and then each of these individual sub function checks should be separate tests. Not sure its worth the time though...

icfaust · 2026-07-20T15:27:59Z

+        if len(args) == 0 and len(kwargs) == 0:
+            return invoke_func(self, *args, **kwargs)

-        if kwargs.get("queue") is not None:


This is removed because we have removed all direct onedal estimator interfaces in sklearnex (spmd?)?

Yeah even in current main branch, support_input_format only wraps host-side sklearn fallback functions, with none accepting a queue, so it's dead code at this point

icfaust

The changes in the active code weren't as extensive as I thought, however the test suite is brutal. We need to have an internal discussion of test hygiene and what we are going to do to make this easier to maintain. I accept a bit of responsibility for having caused this as I sorta pushed get_dataframes_and_queues through a bunch of tests, and now with array API support required, we need to go back and simplify things further. Would like weigh in from others.

good work.

ethanglaser · 2026-07-22T15:02:48Z

/intelci: run

ethanglaser · 2026-07-22T19:29:51Z

Last call for reviews on this, planning to merge by friday

ethanglaser · 2026-07-23T19:01:42Z

/intelci: run

Remove non-array api dpnp support

46eee08

minor follow-up plus examples and tests scope update

22b26ad

revisions on prev non-array api tests

4cee458

ethanglaser and others added 2 commits July 13, 2026 23:45

lingering test fails

67832bb

Merge branch 'uxlfoundation:main' into dev/eglaser-dpnp-array-api-only

3f4cf1b

icfaust reviewed Jul 14, 2026

View reviewed changes

Comment thread sklearnex/utils/_array_api.py

icfaust reviewed Jul 14, 2026

View reviewed changes

Comment thread sklearnex/preview/covariance/covariance.py

icfaust reviewed Jul 14, 2026

View reviewed changes

Comment thread onedal/datatypes/sycl_usm/data_conversion.cpp

icfaust reviewed Jul 14, 2026

View reviewed changes

address some comments and pca spmd

c9fd6b5

fix remaining gpu tests

a6c5701

david-cortes-intel reviewed Jul 15, 2026

View reviewed changes

address comments

839dcfc

expand examples comments and address comment

f5c8e9a

ethanglaser marked this pull request as ready for review July 17, 2026 19:06

ethanglaser requested review from Vika-F, ahuber21, avolkov-intel, maria-Petrova, napetrov, razdoburdin, syakov-intel and yuejiaointel as code owners July 17, 2026 19:06

Copilot AI review requested due to automatic review settings July 17, 2026 19:06

Copilot started reviewing on behalf of ethanglaser July 17, 2026 19:07 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread sklearnex/_device_offload.py

Comment on lines 276 to 286

@@ -303,27 +286,3 @@ def wrapper_impl(*args, **kwargs):

return result

Comment thread sklearnex/tests/test_patching.py

icfaust reviewed Jul 20, 2026

View reviewed changes

icfaust approved these changes Jul 20, 2026

View reviewed changes

Merge branch 'uxlfoundation:main' into dev/eglaser-dpnp-array-api-only

41cf589

This was referenced Jul 20, 2026

isolate dpnp sua offset fix #3330

Merged

force array_api_dispatch for non-numpy array API frameworks #3332

Open

david-cortes-intel reviewed Jul 21, 2026

View reviewed changes

Comment thread sklearnex/preview/preprocessing/_data.py Outdated

ethanglaser and others added 2 commits July 21, 2026 10:46

update transform comment

20c346e

Merge branch 'uxlfoundation:main' into dev/eglaser-dpnp-array-api-only

7252c2c

ethanglaser mentioned this pull request Jul 22, 2026

[maintenance] Simplify wrap_output_data and fix transform_output guard #3248

Merged

7 tasks

resolve merge conflicts

12cab2b

ethanglaser merged commit 65e916d into uxlfoundation:main Jul 23, 2026
30 of 31 checks passed

		@@ -303,27 +286,3 @@ def wrapper_impl(args, *kwargs):
		return result

Uh oh!

Conversation

ethanglaser commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

ethanglaser commented Jul 13, 2026

Uh oh!

ethanglaser commented Jul 13, 2026

Uh oh!

ethanglaser commented Jul 13, 2026

Uh oh!

codecov Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ethanglaser commented Jul 14, 2026

Uh oh!

ethanglaser commented Jul 14, 2026

Uh oh!

david-cortes-intel commented Jul 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

icfaust left a comment

Choose a reason for hiding this comment

Uh oh!

ethanglaser commented Jul 14, 2026

Uh oh!

ethanglaser commented Jul 14, 2026

Uh oh!

ethanglaser commented Jul 14, 2026

Uh oh!

david-cortes-intel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ethanglaser commented Jul 15, 2026

Uh oh!

ethanglaser commented Jul 15, 2026

Uh oh!

david-cortes-intel commented Jul 16, 2026

Uh oh!

ethanglaser commented Jul 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

ethanglaser commented Jul 13, 2026 •

edited

Loading

codecov Bot commented Jul 13, 2026 •

edited

Loading