Geo-Expert

ICML 2026 Workshop AI4Science | Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning

Geo-Expert is a reproducibility-oriented code release for building and evaluating domain-specialized geological reasoning models with parameter-efficient fine-tuning. The repository focuses on the engineering pipeline: cleaning geology Markdown sources, synthesizing source-grounded Chain-of-Thought instruction data, training LoRA adapters, and evaluating models through API, GGUF, or vLLM backends.

The full training corpus, benchmark set, and model checkpoints are not included in this public repository. The scripts are designed to run with local or authorized datasets and model paths.

Highlights

• Three-stage workflow: source extraction, CoT instruction synthesis, and LoRA fine-tuning.
• Lightweight command-line data pipeline for reproducing the Markdown-to-SFT conversion without a GUI.
• Training command templates for Qwen3-8B, Qwen3-32B, and Gemma-3-27B using ms-swift.
• Evaluation runners for OpenAI-compatible APIs, local GGUF models, and vLLM.
• Tiny self-authored examples for checking the public pipeline without private data or paid model calls.

Repository Layout

• data_pipeline/: Markdown cleaning, CoT synthesis, and chat-dataset merge utilities.
• evaluation/: API, GGUF, and vLLM benchmark runners.
• Fine-tuning/: LoRA fine-tuning command templates.
• examples/: tiny self-authored files for checking the public pipeline.
• assets/: figures used by the public README.
• data/: local/private derived datasets, ignored for public release.
• output/: local/private training outputs and checkpoints, ignored for public release.

Installation

Create a Python environment and install dependencies:

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
pip install -r requirements.txt

For the no-cost data-pipeline smoke test, only openai and tqdm are required. Training and local evaluation require heavier dependencies such as torch, transformers, ms-swift, vllm, or llama-cpp-python.

Quick Reproducibility Check

The public examples can be used to verify the complete preprocessing/export path without private data:

python data_pipeline/clean_markdown.py \
  --input-dir examples/sample_md \
  --output-dir /tmp/geo_expert_md_clean

python data_pipeline/build_cot_dataset.py \
  --input-dir /tmp/geo_expert_md_clean \
  --output-dir /tmp/geo_expert_cot_dryrun \
  --max-chunks 3 \
  --dry-run

python data_pipeline/build_cot_dataset.py \
  --input-dir examples/sample_md \
  --output-dir /tmp/geo_expert_cot_export \
  --raw-output examples/sample_raw_generations.jsonl \
  --sft-output /tmp/geo_expert_cot_export/sft_dataset.json \
  --export-only

The final command converts a small prepared teacher-output file into a ShareGPT-style SFT dataset.

Data Preparation

Clean MinerU Markdown exports:

python data_pipeline/clean_markdown.py \
  --input-dir data/md-dataset \
  --output-dir data/md-dataset-clean

Generate source-grounded CoT instruction data from cleaned Markdown. This repository provides a command-line document-to-QA workflow with Geo-Expert prompt templates and filters:

export OPENAI_API_KEY="..."
export OPENAI_BASE_URL="https://api.openai.com/v1"

python data_pipeline/build_cot_dataset.py \
  --input-dir data/md-dataset-clean \
  --output-dir data/CoT-dataset-rebuilt \
  --model DeepSeek-R1 \
  --questions-per-chunk 2 \
  --concurrency 4 \
  --resume

Merge synthesized chat records:

python data_pipeline/merge_json_chat.py \
  --input-dir data/CoT-dataset \
  --input-glob "*SFT.json" \
  --output data/CoT-dataset/merged_chat_dataset.json

Fine-Tuning

The training commands use ms-swift with LoRA. Defaults are aligned with the paper experiments, and paths can be overridden through environment variables. The public repository does not include the private full training corpus, so set DATASET to a local rebuilt or authorized dataset before running:

bash Fine-tuning/train_command/train_Qwen3-8B.sh
bash Fine-tuning/train_command/train_Qwen3-32B.sh
bash Fine-tuning/train_command/train_Gemma-3-27B.sh

Example with local model and output paths:

MODEL_PATH=/path/to/Qwen3-8B \
DATASET=/path/to/merged_chat_dataset.json \
OUTPUT_ROOT=/path/to/output \
CUDA_VISIBLE_DEVICES=0 \
bash Fine-tuning/train_command/train_Qwen3-8B.sh

Evaluation

OpenAI-compatible API benchmark:

export OPENAI_API_KEY="..."
python evaluation/run_benchmark_api.py \
  --input geo_test_dataset.jsonl \
  --output results/deepseek-r1.jsonl \
  --model DeepSeek-R1 \
  --base-url https://llmapi.paratera.com/v1 \
  --concurrency 10

GGUF benchmark:

python evaluation/run_benchmark_gguf.py \
  --input geo_test_dataset.jsonl \
  --output results/gemma3-27b-geo.jsonl \
  --model-path models/gemma3-27b-geo.gguf

vLLM benchmark:

python evaluation/run_benchmark_vllm.py \
  --input geo_test_dataset.jsonl \
  --output results/qwen3-8b.jsonl \
  --model-path Qwen/Qwen3-8B

Notes for Release

The CoT synthesis workflow is implemented as a small independent CLI for document chunking, teacher-model generation, filtering, and ShareGPT export. It was inspired by the open-source Easy Dataset project (https://github.com/ConardLi/easy-dataset, AGPL-3.0), but this repository does not vendor or redistribute Easy Dataset source code.

Do not commit API keys, private checkpoints, full training datasets, or raw copyrighted textbook text. The default .gitignore excludes data/, output/, and common model checkpoint files. For public release, prefer scripts, prompts, model cards, and derived benchmark metadata that you have permission to distribute.

Paper

This repository contains the code release only. The paper is available on OpenReview.