Skip to content

Add SenseBench to Leaderboards - #21

Open
vassiliphilippov wants to merge 1 commit into
PetroIvaniuk:mainfrom
vassiliphilippov:add-sensebench
Open

Add SenseBench to Leaderboards#21
vassiliphilippov wants to merge 1 commit into
PetroIvaniuk:mainfrom
vassiliphilippov:add-sensebench

Conversation

@vassiliphilippov

Copy link
Copy Markdown

Adds one line to Large Language Models (LLMs) and Chatbots → Leaderboards, after MathArena.

- [SenseBench](https://sense-bench.com) - a leaderboard for English word sense disambiguation: models pick the correct WordNet sense for a word in context, scored on a lexicographer-reviewed 4,861-item dataset with every run re-verified from stored raw API responses, by Glite

The section covers general capability, math, code, safety, hallucination and agents, but nothing lexical. SenseBench fills that: each model is shown a target word in its sentence context together with the candidate WordNet 3.0 senses and must return the index of the correct one. 198 verified runs across 63 models so far, alongside classic supervised WSD systems (MFS, BEM, ESCHER, ConSeC) scored on identical items for reference.

Two things that may be of interest given what else is listed here:

  • Submissions are re-verified in CI from the stored raw API request/response artifacts rather than trusted from reported metadata, and results carry bootstrap confidence intervals and paired McNemar comparisons.
  • Cost per million items is reported for both API and self-hosted runs, the latter priced at fixed reference GPU rates so runs stay comparable across rentals.

Results are also available as JSON at https://sense-bench.com/data/leaderboard.json. Only README.md is touched, matching the recent merged PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant