Skip to content

Warn when the calibration set is too small for the chosen confidence level - #50

Open
shivamlalakiya wants to merge 1 commit into
henrikbostrom:mainfrom
shivamlalakiya:fix/classifier-calibration-warning
Open

Warn when the calibration set is too small for the chosen confidence level#50
shivamlalakiya wants to merge 1 commit into
henrikbostrom:mainfrom
shivamlalakiya:fix/classifier-calibration-warning

Conversation

@shivamlalakiya

Copy link
Copy Markdown

Fixes #47.

ConformalRegressor.predict_int (lines 912, 930) and ConformalPredictiveSystem.predict (2387, 2419, 2440, 2484) warn when the calibration set is too small for the requested confidence level and the output degenerates to maximum size. ConformalClassifier has the same failure mode and no warning.

With smoothing=False the smallest attainable p-value is 1/(n_cal+1), so when 1/(n_cal+1) >= 1-confidence every class clears the threshold for every test object and every prediction set contains every label. At confidence=0.95 this is the same 19-example boundary the regressor already warns about, and the existing message reads correctly for the classifier with "sets" in place of "intervals".

This is the failure mode that prompted the v0.2.0 warnings after @Geethen reported it for the regressor.

What this adds

check_calibration_size, placed after get_classification_results and mirroring the wording of both existing regressor warnings (plain and Mondrian), called from the batch methods that produce prediction sets:

  • ConformalClassifier.predict_set
  • ConformalClassifier.evaluate (batch branch)
  • WrapClassifier.evaluate (batch branch)

Scope choices

  • Batch only. The existing regressor and CPS warnings sit on the batch methods; the online methods on both sides are silent. Warning on the classifier online paths would introduce a new asymmetry, so I left them alone.
  • No-op when smoothing=True. Smoothed p-values can fall arbitrarily close to zero regardless of n_cal, so there is nothing to warn about on the default path.
  • Mondrian reports only the bins present in the test set. This keeps the class-conditional case to one warning per undersized class rather than one per class per call, since WrapClassifier.predict_set calls cc.predict_set once per class.
  • The batch paths of WrapClassifier.predict_set, including the class-conditional one, route through ConformalClassifier.predict_set and are covered by that call rather than a separate one.

Happy to change any of these — particularly the batch/online scope, if you would rather have the online paths covered too.

Checked

Script exercising the plain, Mondrian, class-conditional and WrapClassifier paths, all passing on this branch:

  • n_cal=18, non-smoothed: returns [[1, 1]] and warns; n_cal=19: no warning
  • n_cal=18, smoothed: no warning
  • Mondrian with bins of 18 and 100: only the undersized bin is named
  • WrapClassifier with class_cond=True and 15 per class: fully vacuous sets, one warning per class; evaluate warns on the same setup
  • default smoothed paths unchanged and silent

I did not add it to the repository since there is no test suite to add it to, but glad to include it if useful.

ConformalRegressor.predict_int and ConformalPredictiveSystem.predict warn
when the calibration set is too small for the chosen confidence level and
the output degenerates to maximum size. ConformalClassifier had no such
warning, although it has the same failure mode: with non-smoothed p-values
the smallest attainable p-value is 1/(n_cal+1), so when
1/(n_cal+1) >= 1-confidence every class clears the threshold and every
prediction set contains every label.

Add check_calibration_size, mirroring the wording of the existing
regressor warnings, and call it from the batch methods that produce
prediction sets: ConformalClassifier.predict_set,
ConformalClassifier.evaluate and WrapClassifier.evaluate. The batch paths
of WrapClassifier.predict_set, including the class-conditional one, route
through ConformalClassifier.predict_set and are covered by that call.

Smoothed p-values can fall arbitrarily close to zero regardless of n_cal,
so the check is a no-op when smoothing=True. Under Mondrian calibration
only the bins present in the test set are reported, which keeps the
class-conditional case to one warning per undersized class.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ConformalClassifier gives no warning when the calibration set is too small for the confidence level

1 participant