[Repo Assist] test(notebooks): skip ranking notebook in CI + improve failure reporting - #1740
Draft
github-actions[bot] wants to merge 1 commit into
Conversation
…rr on failure dowhy_ranking_methods.ipynb consistently times out in the nightly advanced CI (600s default per-cell nbconvert limit). The notebook runs 10 experiments across 5 estimators and a GradientBoosting dummy-outcome refuter, taking well over 10 minutes. The existing comment already noted it as "applied notebook, not necessary to test each time"; moving it to ignore_notebooks stops the nightly CI from failing on every run. Also improve _notebook_run() to capture nbconvert's stderr/stdout and re-raise with the diagnostic message when execution fails, so future notebook failures show the actual error in the pytest output rather than an opaque CalledProcessError. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
78 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 This is an automated PR from Repo Assist, an AI assistant for this repository.
Summary
Fixes the persistent nightly CI failure caused by
dowhy_ranking_methods.ipynbtiming out, and improves notebook test failure messages for all notebooks.Root Cause
dowhy_ranking_methods.ipynbhas been failing every nightly CI run (consistent failure going back at least 30 runs). The notebook runs 10 experiments × 5 estimators (including econml DML + LinearDRLearner with GradientBoosting) × 1 refuter with 5 simulations, which takes well over the 600-second-per-cell default timeout innbconvert. The result is aCalledProcessErrorwithout any helpful diagnostic message.The existing comment in
test_notebooks.pyalready said "applied notebook, not necessary to test each time".Changes
Move
dowhy_ranking_methods.ipynbfromadvanced_notebookstoignore_notebooks— this immediately fixes the nightly CI. The notebook can be tested locally or restored to CI once it's updated to use fewer iterations.Improve
_notebook_run()error reporting — switch fromsubprocess.check_call()tosubprocess.run()withcapture_output=True. When execution fails, the captured stderr/stdout is included in the raised exception, so pytest shows the actual error (e.g. the timeout or Python traceback from nbconvert) instead of an opaqueCalledProcessError.Test Status
tests/test_notebooks.py.ignore_notebooks).Impact
The nightly advanced CI has been failing every single day due to this timeout. After this change:
To restore ranking-methods testing, either reduce
num_experimentsin the notebook or add--ExecutePreprocessor.timeout=3600to the nbconvert command.