fix(app): avoid KeyError when no filter OP provides tunable stats - #1073
Merged
Merged
Conversation
Visualize.filter_dataset collects one boolean mask per filter OP listed in op_stats_dict, which covers 13 of the 58 registered filters (none of the image/video/audio ones). When a recipe only uses uncovered filters, `conds` stays empty and np.all([], axis=0) collapses to the scalar np.True_; dataframe.loc[np.True_] then looks up a label instead of applying a mask and raises KeyError: True, which is the traceback in issue datajuicer#872. Combine the masks through combine_conds(), which returns an all-True mask of the right length when there is nothing to filter on, and cover it with a unit test that loads app.py without a Streamlit runtime. Fixes datajuicer#872
cmgzn
approved these changes
Sep 22, 2026
cmgzn
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, Thanks for your work
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #872 (assigned to @HYLcool and @Dludora since January without a linked branch — if a fix is already planned, feel free to close this).
Problem
app.py::Visualize.filter_datasetbuilds one boolean mask per filter OP whose stats keys are listed inop_stats_dict. That dict covers 13 of the 58 registered filters — none of the image/video/audio filters, nor newer text filters such asgeneral_field_filter. With a recipe whose filters are all uncovered (e.g. onlyimage_shape_filter, the reporter's case),condsstays empty, and with "Process by op order" unchecked (the default):Reproduced by executing the real
app.py(Streamlit stubbed) with a recipe containing onlyimage_shape_filter; the pandas frames in the traceback match #872 line for line (pandas 2.2.3, the locked version; also fails on 3.0). The bundled default recipe (demos/process_simple/process.yaml,language_id_score_filter) is covered, so this needs a user recipe — but any multimodal recipe is.Change
Move the mask combination into a module-level
combine_conds(conds, num_samples): the non-empty branch keeps the exactnp.all([...], axis=0)expression; the empty case returnsnp.ones(num_samples, dtype=bool)("nothing to filter on ⇒ keep everything"), mirroring therange(len(stats))shape theorderedbranch already uses. Theordered=Truebranch andnp.invert(all_conds)for the discarded view are unchanged.Tests
tests/tools/test_app.pyloadsapp.pyby path (astests/tools/test_build_op_doc.pydoes for the root-level hook script) and checks: emptycondsyields an all-True mask thatdataframe.locaccepts; non-emptycondsstill combine element-wise; the inverted mask selects nothing. Fails onmain(KeyError: True), passes here.black/isort/flake8with the versions pinned in.pre-commit-config.yamlpass onapp.py(the tests directory is excluded from pre-commit). Note for CI: partial mode mapsapp.pyto a root-leveltest_app.pythat does not exist, so this PR will fall back to the full test run.Written with Claude Code (AI-assisted) and submitted under the account owner's authorization.