WASSCE Benchmark for Open Language Models

Evaluating LLMs against the exam 2 million West Africans take every year.

---

What Is This?

A benchmark for evaluating open language models against the West African Senior School Certificate Examination (WASSCE) -- the standardized exam administered across Ghana, Nigeria, Sierra Leone, The Gambia, and Liberia.

Most LLM benchmarks (MMLU, GPQA, HumanEval) draw from American and European academic traditions. This project asks a different question: how well do open models perform on knowledge that matters to the rest of the world?

The benchmark uses the Situated Evaluation Framework -- testing both universal knowledge (math, basic science) and regionally situated knowledge (West African governance, local ecosystems, cultural context) to measure where models actually have gaps.

Why WASSCE?

• Scale: 2+ million candidates annually across 5 countries
• Stakes: Gates university admission for an entire region
• Knowledge gradient: Ranges from universal (algebra, physics) to deeply regional (ECOWAS policy, Ghanaian constitutional law)
• Underrepresented: No existing LLM benchmark covers this content

Subjects

Subject	Type	What It Tests
Core Mathematics	Universal (Control)	Algebra, geometry, trigonometry, statistics
English Language	Cultural Fluency	Comprehension, vocabulary, grammar with West African literary context
Integrated Science	Applied Regional	Biology, chemistry, physics with local agricultural/health context
Social Studies	Situated (Stress Test)	West African governance, ECOWAS, Ghanaian constitution, regional development

Quick Start

1. Install

git clone https://github.com/YOUR_USERNAME/wassce-benchmark.git
cd wassce-benchmark
pip install -r requirements.txt

2. Configure

Copy the example config and add your API key:

cp configs/crusoe_example.yaml configs/crusoe.yaml

Edit configs/crusoe.yaml and add your Crusoe Intelligence Foundry bearer token. Generate one at console.crusoecloud.com/foundry/api-keys.

3. Run the Eval

# Run all subjects against a single model
python eval.py --model deepseek-ai/DeepSeek-V3-0324 --config configs/crusoe.yaml

# Run a specific subject
python eval.py --model deepseek-ai/DeepSeek-V3-0324 --subject core_maths --config configs/crusoe.yaml

# Run all models (uses all models defined in config)
python eval.py --all-models --config configs/crusoe.yaml

4. View Results

# Print a summary table
python scripts/summarize.py results/

# Export to CSV
python scripts/summarize.py results/ --format csv --output results/summary.csv

Supported Models

All models are available via Crusoe Intelligence Foundry managed inference.

Model	Archetype	Model ID
DeepSeek V3	Dense Baseline	deepseek-ai/DeepSeek-V3-0324
Llama 3.3 70B	Dense Baseline	meta-llama/Llama-3.3-70B-Instruct
GPT-OSS 120B	Dense Baseline	openai/gpt-oss-120b
DeepSeek R1	Reasoning Specialist	deepseek-ai/DeepSeek-R1-0528
Kimi K2 Thinking	Reasoning Specialist	moonshotai/Kimi-K2-Thinking
Nemotron 3 Nano 30B	Efficient (MoE)	nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B
Gemma 3 12B	Efficient	google/gemma-3-12b-it
Qwen3 235B	Multilingual/Large	Qwen/Qwen3-235B-A22B-Instruct-2507
Nemotron 3 Super 120B	Large (MoE)	nvidia/NVIDIA-Nemotron-3-Super-120B-A12B

You can add any OpenAI-compatible model by editing the config file.

Dataset Format

Each question is stored as a JSON object:

{
  "id": "core_maths_2023_001",
  "subject": "core_maths",
  "year": 2023,
  "question_number": 1,
  "question": "Simplify 3(2x - 4) - 2(x - 5)",
  "options": {
    "A": "4x - 2",
    "B": "4x - 22",
    "C": "4x + 2",
    "D": "4x + 22"
  },
  "correct_answer": "A",
  "topic": "algebra"
}

Questions are organized by subject in data/{subject}/questions.json.

Project Structure

wassce-benchmark/
  configs/
    crusoe_example.yaml     # Example config (copy and add your API key)
  data/
    core_maths/
      questions.json         # Core Maths question bank
    english/
      questions.json         # English Language question bank
    integrated_science/
      questions.json         # Integrated Science question bank
    social_studies/
      questions.json         # Social Studies question bank
  scripts/
    summarize.py             # Results aggregation and export
    validate_dataset.py      # Dataset integrity checks
  results/                   # Model evaluation results (gitignored, except samples)
  eval.py                    # Main evaluation script
  requirements.txt           # Python dependencies

Contributing

Adding Questions

The most valuable contribution is expanding the dataset. If you have access to WASSCE past questions:

1. Format questions according to the JSON schema above
2. Place them in the appropriate data/{subject}/ directory
3. Run python scripts/validate_dataset.py to verify formatting
4. Open a PR

Important: Only submit questions from publicly available past papers. Do not submit questions from unreleased or current exam papers.

Adding Models

To benchmark a new model:

1. Add the model config to your configs/crusoe.yaml (or create a new config for a different provider)
2. Run the eval
3. Open a PR with your results in results/

Reporting Issues

If you find a question with an incorrect answer key, open an issue with the question ID and the correct answer with a source.

The Situated Evaluation Framework

This benchmark implements the Situated Evaluation Framework -- a methodology for building custom LLM evaluations that test both universal and domain-specific knowledge.

The core principle: include universal subjects (control group, expected to perform well) alongside situated subjects (stress test, performance depends on training data representation). The spread between them reveals whether a model has a general capability problem or a training data problem.

This framework is portable. If you want to build a similar benchmark for a different exam or domain, the methodology and tooling in this repo are designed to be adapted.

Read the full writeup: How to Build Your Own LLM Benchmark

License

MIT License. See LICENSE for details.

The dataset contains questions derived from publicly available WASSCE past examination papers. These questions are used for research and educational evaluation purposes.

---

Built by Emmanuel Acheampong | Powered by Crusoe Intelligence Foundry