Skip to content

fix(app): avoid KeyError when no filter OP provides tunable stats - #1073

Merged
cmgzn merged 1 commit into
datajuicer:mainfrom
Linxiushen:fix-app-empty-filter-conds
Sep 22, 2026
Merged

cmgzn merged 1 commit into
datajuicer:mainfrom
Linxiushen:fix-app-empty-filter-conds

Conversation

@Linxiushen

Copy link
Copy Markdown
Contributor

Fixes #872 (assigned to @HYLcool and @Dludora since January without a linked branch — if a fix is already planned, feel free to close this).

Problem

app.py::Visualize.filter_dataset builds one boolean mask per filter OP whose stats keys are listed in op_stats_dict. That dict covers 13 of the 58 registered filters — none of the image/video/audio filters, nor newer text filters such as general_field_filter. With a recipe whose filters are all uncovered (e.g. only image_shape_filter, the reporter's case), conds stays empty, and with "Process by op order" unchecked (the default):

all_conds = np.all([list(cond.values())[0] for cond in conds], axis=0)   # np.all([], axis=0) -> np.True_
...
examples = dataframe.loc[cond]                                             # label lookup, not a mask
KeyError: True

Reproduced by executing the real app.py (Streamlit stubbed) with a recipe containing only image_shape_filter; the pandas frames in the traceback match #872 line for line (pandas 2.2.3, the locked version; also fails on 3.0). The bundled default recipe (demos/process_simple/process.yaml, language_id_score_filter) is covered, so this needs a user recipe — but any multimodal recipe is.

Change

Move the mask combination into a module-level combine_conds(conds, num_samples): the non-empty branch keeps the exact np.all([...], axis=0) expression; the empty case returns np.ones(num_samples, dtype=bool) ("nothing to filter on ⇒ keep everything"), mirroring the range(len(stats)) shape the ordered branch already uses. The ordered=True branch and np.invert(all_conds) for the discarded view are unchanged.

Tests

tests/tools/test_app.py loads app.py by path (as tests/tools/test_build_op_doc.py does for the root-level hook script) and checks: empty conds yields an all-True mask that dataframe.loc accepts; non-empty conds still combine element-wise; the inverted mask selects nothing. Fails on main (KeyError: True), passes here.

black / isort / flake8 with the versions pinned in .pre-commit-config.yaml pass on app.py (the tests directory is excluded from pre-commit). Note for CI: partial mode maps app.py to a root-level test_app.py that does not exist, so this PR will fall back to the full test run.

Written with Claude Code (AI-assisted) and submitted under the account owner's authorization.

Visualize.filter_dataset collects one boolean mask per filter OP listed
in op_stats_dict, which covers 13 of the 58 registered filters (none of
the image/video/audio ones). When a recipe only uses uncovered filters,
`conds` stays empty and np.all([], axis=0) collapses to the scalar
np.True_; dataframe.loc[np.True_] then looks up a label instead of
applying a mask and raises KeyError: True, which is the traceback in
issue datajuicer#872.

Combine the masks through combine_conds(), which returns an all-True
mask of the right length when there is nothing to filter on, and cover
it with a unit test that loads app.py without a Streamlit runtime.

Fixes datajuicer#872

@cmgzn cmgzn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thanks for your work

@cmgzn
cmgzn merged commit 70e8950 into datajuicer:main Sep 22, 2026
4 checks passed

This branch was successfully deployed

1 active deployment
Testing — a19c09f7 Deployed Sep 21, 2026 by Linxiushen via unittest-single #1762
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

process后根据图片路径无法加载

2 participants