Skip to content
BranchesTags

Repository files navigation

Surgical Instrument Tracking – CholecTrack20

A complete pipeline for multi-object detection and tracking of laparoscopic surgical instruments using the CholecTrack20 dataset, YOLOv10, BoT-SORT, and a custom Memory Bank for out-of-body re-identification.

Project Structure

Forschungsprojekt/
├── requirements.txt       # Python dependencies
├── prepare_data.py        # JSON → YOLO labels, image symlinks
├── cholectrack20.yaml     # Ultralytics dataset config
├── train.py               # Multi-GPU YOLOv10 training
├── track.py               # Inference + BoT-SORT + Memory Bank ReID
├── README.md              # This file
│
├── logs/                  # nohup output logs (created on first run)
│   ├── prepare_data_trainval.log
│   ├── prepare_data_test.log
│   ├── train.log
│   ├── track_VID01.log
│   └── ...
│
├── data/                  # Generated by prepare_data.py
│   ├── images/
│   │   ├── train/         # Symlinks to scratch3 PNG frames
│   │   ├── val/           # Symlinks to scratch3 PNG frames
│   │   └── test/          # Extracted frames from .mp4 (test split)
│   └── labels/
│       ├── train/         # YOLO-format .txt label files
│       ├── val/
│       └── test/
│
└── runs/                  # Created by Ultralytics during training
    └── detect/
        └── cholectrack20/
            └── weights/
                ├── best.pt
                └── last.pt

Instrument Classes

ID Name Description
0 grasper Most common; can appear ≥2 at once
1 bipolar Cauterisation forceps
2 hook Electrosurgical hook
3 scissors Curved / straight micro-scissors
4 clipper Clip applier (haemostatic clips)
5 irrigator Irrigation / suction cannula
6 specimen_bag Endobag / specimen retrieval bag

Step-by-Step Execution

0. Prerequisites – Hardware & Software

  • 3× NVIDIA RTX 4090 (24 GB VRAM each) on a shared cluster
  • CUDA ≥ 12.1, cuDNN ≥ 8.9
  • Python ≥ 3.10 (cluster has Python 3.13.11 – confirmed compatible)

1. Create & Activate Virtual Environment

# Navigate to the project root
cd /graphics/scratch2/students/eichler/Forschungsprojekt

# Create a Python 3.10+ virtual environment
python3 -m venv .venv

# Activate it (must be done in every new terminal session)
source .venv/bin/activate

# Verify Python version
python --version

Note: All subsequent commands assume the virtual environment is activated.

2. Install Dependencies

# Upgrade pip first
pip install --upgrade pip setuptools wheel

# Step 1: install everything except boxmot
pip install -r requirements.txt

# Step 2: install boxmot WITHOUT dependency resolution
# (boxmot 12.x metadata pins numpy==1.26.4 which has no cp313 wheel;
#  its runtime code works fine with numpy 2.x that was installed above)
pip install boxmot --no-deps

# Optional: verify GPU access
python -c "import torch; print(torch.cuda.device_count(), 'GPU(s) found')"
# Expected output:  3 GPU(s) found

Troubleshoot: If torch cannot find CUDA, install the correct CUDA-specific wheel:

pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124

Hinweis zu setsid: Alle langen Prozesse werden mit setsid python ... gestartet. setsid erstellt eine neue Session ohne kontrollierende TTY, so dass kein SIGHUP beim SSH-Disconnect gesendet wird – weder an den Hauptprozess noch an gespawnte Subprozesse (z. B. torchrun beim DDP-Training).

3. Prepare Data (Labels + Symlinks)

prepare_data.py reads the read-only CholecTrack20 JSON annotations, converts bounding boxes to YOLO format, writes them as .txt label files in your working directory, and creates symbolic links for the images so that no data is copied.

3a. Convert training and validation splits

mkdir -p logs
setsid python prepare_data.py \
    --dataset_root /graphics/scratch3/datasets/cholectrack20 \
    --work_dir     /graphics/scratch2/students/eichler/Forschungsprojekt \
    --splits       train val \
    > logs/prepare_data_trainval.log 2>&1 &
echo "PID $! – tail -f logs/prepare_data_trainval.log"

Follow progress:

tail -f logs/prepare_data_trainval.log

Expected console output (abbreviated):

Processing split: 'train'  (Training)
  [VID02] frames= 2545  boxes=  3812  skipped=   0
  [VID04] frames= 2565  boxes=  4100  skipped=   0
  ...
Processing split: 'val'  (Validation)
  [VID110] frames= 2421  boxes=  3500  skipped=   0
  [VID30]  frames= 1800  boxes=  2700  skipped=   0
...
Total frames processed : <N>
Total boxes written    : <M>
Done.

3b. (Optional) Prepare test split

The test split has no pre-extracted Frames – the script decodes them from .mp4:

mkdir -p logs
setsid python prepare_data.py \
    --dataset_root /graphics/scratch3/datasets/cholectrack20 \
    --work_dir     /graphics/scratch2/students/eichler/Forschungsprojekt \
    --splits       test \
    --extract_test_fps 1 \
    > logs/prepare_data_test.log 2>&1 &
echo "PID $! – tail -f logs/prepare_data_test.log"

Follow progress:

tail -f logs/prepare_data_test.log

Disk space: Each extracted frame is ~300–500 KB (PNG).
8 test videos × ~2 500 frames each ≈ 10–20 GB.

3c. Verify symlinks

# Count label and image files per split
echo "Train images: $(ls data/images/train | wc -l)"
echo "Train labels: $(ls data/labels/train | wc -l)"
echo "Val   images: $(ls data/images/val   | wc -l)"
echo "Val   labels: $(ls data/labels/val   | wc -l)"

# Verify that symlinks resolve correctly
ls -la data/images/train | head -5

4. Multi-GPU Training

4a. Run training (DDP on 3× RTX 4090)

mkdir -p logs
setsid python train.py \
    --model   yolov10x.pt \
    --data    cholectrack20.yaml \
    --epochs  100 \
    --imgsz   640 \
    --batch   48 \
    --devices 0,1,2 \
    --workers 8 \
    --project runs/detect \
    --name    cholectrack20 \
    > logs/train.log 2>&1 &
echo "PID $! – tail -f logs/train.log"

Follow progress:

tail -f logs/train.log

Argument reference:

Argument Default Description
--model yolov10x.pt Pre-trained YOLO weights (auto-downloaded)
--data cholectrack20.yaml Dataset config
--epochs 500 Total epochs
--imgsz 640 Input resolution (px²)
--batch 48 Global batch (16 per GPU)
--devices 0,1,2 GPU device IDs
--workers 8 DataLoader workers per GPU
--patience 50 Early-stopping epochs
--fl_gamma 2.0 Focal loss gamma (higher = more focus on rare/hard classes)
--lr0 0.01 Initial learning rate
--lrf 0.001 Final LR (fraction of lr0)
--warmup_epochs 3 Cosine warmup duration
--name cholectrack20 Sub-folder inside --project
--resume (none) Resume from a checkpoint

4b. Resume interrupted training

setsid python train.py \
    --resume  runs/detect/cholectrack20/weights/last.pt \
    --devices 0,1,2 \
    > logs/train_resume.log 2>&1 &
echo "PID $! – tail -f logs/train_resume.log"

4c. Monitor training

# Watch training metrics (Ultralytics writes CSV + PNG plots)
watch -n 30 "tail -5 runs/detect/cholectrack20/results.csv"

# Or open TensorBoard (if tensorboard is installed)
tensorboard --logdir runs/detect/cholectrack20

4d. Expected results

After 100 epochs on 3× RTX 4090, expect:

Metric Approx. target
mAP50 0.70 – 0.82
mAP50-95 0.50 – 0.65
Training time 8 – 14 hours

Tip: Use --model yolov10l.pt for faster iterations during debugging.

5. Tracking & Evaluation

5a. Download BoT-SORT ReID weights

# boxmot can download them automatically on first run, or manually:
python -c "
from boxmot import BoTSORT
from pathlib import Path
# triggers automatic download of osnet_x0_25_msmt17.pt
tracker = BoTSORT(reid_weights=Path('osnet_x0_25_msmt17.pt'), device='cuda:0')
print('ReID model ready:', Path('osnet_x0_25_msmt17.pt').exists())
"

5b. Run tracking on a test video

mkdir -p logs
setsid python track.py \
    --weights   runs/detect/cholectrack20/weights/best.pt \
    --source    /graphics/scratch3/datasets/cholectrack20/Testing/VID01/VID01.mp4 \
    --out_dir   results/VID01 \
    --device    0 \
    --conf      0.25 \
    --iou       0.45 \
    --save_video \
    > logs/track_VID01.log 2>&1 &
echo "PID $! – tail -f logs/track_VID01.log"

Follow progress:

tail -f logs/track_VID01.log

Argument reference:

Argument Default Description
--weights (required) Path to best.pt
--source (required) .mp4 video or image folder
--out_dir results Output directory
--device 0 Single GPU for inference
--conf 0.25 Detection confidence threshold
--iou 0.45 NMS IoU threshold
--reid_weights osnet_x0_25_msmt17.pt BoT-SORT appearance model
--trocar_margin 0.12 Border fraction for trocar zone
--match_thresh 0.72 Histogram similarity for Memory Bank ReID
--max_age 600 Memory Bank expiry (frames)
--save_video True Write annotated video

Outputs in results/VID01/:

  • VID01_tracked.mp4 – annotated video with bounding boxes and track IDs
  • VID01.txt – CholecTrack20 evaluation format: frameid,-1,x_norm,y_norm,w_norm,h_norm,score,class_id
  • VID01_summary.json – tracking statistics (total detections, unique IDs, Memory Bank remaps)

5c. Run tracking on all test videos (batch)

Runs all 8 test videos sequentially in the background, one log file per video:

mkdir -p logs
for VID in VID01 VID06 VID07 VID111 VID12 VID25 VID39 VID92; do
    echo "=== Queuing $VID ==="
    setsid python track.py \
        --weights  runs/detect/cholectrack20/weights/best.pt \
        --source   /graphics/scratch3/datasets/cholectrack20/Testing/${VID}/${VID}.mp4 \
        --out_dir  results/${VID} \
        --device   0 \
        --conf     0.25 \
        --save_video \
        > logs/track_${VID}.log 2>&1
    echo "  done → results/${VID}/  (log: logs/track_${VID}.log)"
done &
echo "Batch PID $!"

Watch a specific video's progress:

tail -f logs/track_VID06.log

Check all logs at once:

tail -n 3 logs/track_*.log

5d. Quick validation detection

# Run YOLO val (detection mAP only, no tracking)
mkdir -p logs
setsid python -c "
from ultralytics import YOLO
m = YOLO('runs/detect/cholectrack20/weights/best.pt')
r = m.val(data='cholectrack20.yaml', imgsz=640, device=0)
print('mAP50:', r.box.map50)
print('mAP50-95:', r.box.map)
" > logs/val.log 2>&1 &
echo "PID $! – tail -f logs/val.log"

Design Notes

Augmentation Strategy

To make the detector robust to laparoscopic-specific artefacts, two levels of augmentation are applied:

  1. YOLO built-in parameters (applied at the mosaic/copy-paste level):

    • Mosaic + copy-paste → rare instrument class oversampling
    • Geometric: degrees=10, scale=0.5, shear=2, perspective=0.0001
    • Colour: hsv_s=0.7, hsv_v=0.4

  2. Albumentations pipeline (injected via Ultralytics callback):

    • RandomFog → surgical smoke / CO₂ insufflation haze
    • MotionBlur + GaussianBlur → camera shake, fast instrument movement
    • RandomBrightnessContrast + CLAHE → variable cavity illumination
    • HueSaturationValue + RGBShift → blood and bile colour casts
    • GaussNoise → camera sensor noise
    • CoarseDropout → partial occlusion by tissue

Memory Bank Re-Identification

The MemoryBank in track.py maintains intraoperative trajectories:

Instrument exits near border
        │
        ▼
   Save HSV colour histogram + class to MemoryBank
   (keyed by canonical track ID)
        
Instrument re-enters near border (new BoT-SORT ID)
        │
        ▼
   Compute HSV histogram of new detection
        │
        ▼
   Compare with bank entries of same class
        │
   correlation > 0.72?
        │                │
       Yes               No
        │                │
  Restore old ID    Keep new ID

The histogram comparison uses OpenCV's compareHist with HISTCMP_CORREL.

Troubleshooting

Problem Solution
CUDA out of memory Reduce --batch (e.g. --batch 24) or use --model yolov10l.pt
FileNotFoundError: osnet...pt Run the ReID download cell in §5a
Symlinks not resolving Check ls -la data/images/train – source must be readable
Low mAP on grasper Increase mosaic=1.0 and copy_paste=0.3; class is present in nearly every frame
Tracking ID fragmentation Lower --trocar_margin or tune --match_thresh
boxmot import error pip install boxmot>=10.0.0 lapx>=0.5.5

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages