Warn when the calibration set is too small for the chosen confidence level - #50
Open
shivamlalakiya wants to merge 1 commit into
Open
Conversation
ConformalRegressor.predict_int and ConformalPredictiveSystem.predict warn when the calibration set is too small for the chosen confidence level and the output degenerates to maximum size. ConformalClassifier had no such warning, although it has the same failure mode: with non-smoothed p-values the smallest attainable p-value is 1/(n_cal+1), so when 1/(n_cal+1) >= 1-confidence every class clears the threshold and every prediction set contains every label. Add check_calibration_size, mirroring the wording of the existing regressor warnings, and call it from the batch methods that produce prediction sets: ConformalClassifier.predict_set, ConformalClassifier.evaluate and WrapClassifier.evaluate. The batch paths of WrapClassifier.predict_set, including the class-conditional one, route through ConformalClassifier.predict_set and are covered by that call. Smoothed p-values can fall arbitrarily close to zero regardless of n_cal, so the check is a no-op when smoothing=True. Under Mondrian calibration only the bins present in the test set are reported, which keeps the class-conditional case to one warning per undersized class.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #47.
ConformalRegressor.predict_int(lines 912, 930) andConformalPredictiveSystem.predict(2387, 2419, 2440, 2484) warn when the calibration set is too small for the requested confidence level and the output degenerates to maximum size.ConformalClassifierhas the same failure mode and no warning.With
smoothing=Falsethe smallest attainable p-value is1/(n_cal+1), so when1/(n_cal+1) >= 1-confidenceevery class clears the threshold for every test object and every prediction set contains every label. Atconfidence=0.95this is the same 19-example boundary the regressor already warns about, and the existing message reads correctly for the classifier with "sets" in place of "intervals".This is the failure mode that prompted the v0.2.0 warnings after @Geethen reported it for the regressor.
What this adds
check_calibration_size, placed afterget_classification_resultsand mirroring the wording of both existing regressor warnings (plain and Mondrian), called from the batch methods that produce prediction sets:ConformalClassifier.predict_setConformalClassifier.evaluate(batch branch)WrapClassifier.evaluate(batch branch)Scope choices
smoothing=True. Smoothed p-values can fall arbitrarily close to zero regardless ofn_cal, so there is nothing to warn about on the default path.WrapClassifier.predict_setcallscc.predict_setonce per class.WrapClassifier.predict_set, including the class-conditional one, route throughConformalClassifier.predict_setand are covered by that call rather than a separate one.Happy to change any of these — particularly the batch/online scope, if you would rather have the online paths covered too.
Checked
Script exercising the plain, Mondrian, class-conditional and
WrapClassifierpaths, all passing on this branch:n_cal=18, non-smoothed: returns[[1, 1]]and warns;n_cal=19: no warningn_cal=18, smoothed: no warningWrapClassifierwithclass_cond=Trueand 15 per class: fully vacuous sets, one warning per class;evaluatewarns on the same setupI did not add it to the repository since there is no test suite to add it to, but glad to include it if useful.