[Reproducibility Repository] MOEns: Post-Hoc Multi-Objective Ensembling for Predictive Accuracy and Adversarial Robustness in AutoML

This repository is related to the bachelor thesis "Exploring Post-Hoc Multi-Objective Ensembling for Predictive Accuracy and Adversarial Robustness in AutoML". It provides the code used to generate Auto-Sklearn base model pools, run post-hoc ensembling methods, and evaluate ensembles with respect to clean predictive performance and adversarial robustness.

The repository builds on the existing Q(D)O-ES codebase by Purucker et al. (2023). The original framework is used for running Auto-Sklearn, storing model-pool data, pruning base models, and executing post-hoc ensemble selection methods. This repository extends the framework with MOEns, a multi-objective post-hoc ensembling method that optimises continuous ensemble weights with NSGA-II.

We ran the experiments in a parallel HPC environment. The repository provides scripts and example commands, but users may need to adapt paths, environments, and parallelisation scripts to their own system.

Thesis TL;DR

MOEns is a post-hoc multi-objective ensembling method for AutoML that jointly considers clean predictive performance and adversarial robustness.

Thesis Abstract

Automated Machine Learning systems generate heterogeneous pools of candidate models that are commonly combined through post-hoc ensembling. Existing methods, including Greedy Ensemble Selection, CMA-ES-based weight optimisation, QO-ES and QDO-ES, mainly optimise predictive performance on clean validation data. However, high clean performance does not necessarily imply robustness against adversarial perturbations.

This thesis investigates whether post-hoc ensembling can improve adversarial robustness in AutoML. It proposes a multi-objective ensembling method, MOEns, that optimises continuous ensemble weight vectors using NSGA-II with respect to clean predictive performance and a surrogate robustness objective. Selected Pareto-optimal ensembles are then re-evaluated using PermuteAttack to estimate their true adversarial robustness.

Experiments on tabular classification datasets with Auto-Sklearn base model pools show that the robustness of individual base models varies substantially across datasets and classification algorithms. Compared to the single-objective baseline QO-ES, MOEns achieves slightly lower clean balanced accuracy, but improves balanced adversarial accuracy on most datasets and often shows more stable robustness across folds. The Pareto-front analysis further shows that MOEns can reveal meaningful trade-offs between clean performance and adversarial robustness, although the surrogate robustness objective remains imperfect.

Results Teaser

The goal of post-hoc ensembling in this repository is to construct ensembles from a pool of base models generated by Auto-Sklearn. MOEns extends this setting by optimising two objectives:

1. clean predictive performance on validation data, and
2. surrogate adversarial robustness based on adversarial examples generated with PermuteAttack.

After the NSGA-II optimisation, selected Pareto-optimal ensembles are re-attacked with PermuteAttack to obtain true attack-based robustness values.

In the thesis experiments, MOEns was compared against QO-ES as a single-objective population-based baseline. The main findings were:

• MOEns achieved higher balanced adversarial accuracy than QO-ES on most evaluated datasets.
• MOEns often showed more stable adversarial robustness across folds.
• QO-ES generally retained slightly higher clean balanced accuracy.
• The Pareto fronts revealed a visible trade-off between clean predictive performance and surrogate-predicted robustness.
• The surrogate robustness objective provided a useful but imperfect signal and did not always match the true PermuteAttack-based robustness evaluation.

Data

The repository contains a minimal example under benchmark/input/minimal_example_ens. For full experiments, base model pools must be generated with the scripts in assembled_ask or provided in the expected metatask format.

The thesis experiments used OpenML classification datasets, Auto-Sklearn base model pools, SiloTopN pruning to 50 base models, and folds 0--2 for computational feasibility.

Content Overview

• assembled_ask: Code for running Auto-Sklearn, collecting the generated base models, and building metatasks.
• assembled_ensembles: Code for running post-hoc ensembling methods, including GES, QO-ES, QDO-ES, and MOEns.
• assembled_ensembles/methods/moo: Implementation of MOEns, the NSGA-II optimisation problem, and the PermuteAttack-based robustness evaluation.
• benchmark: Setup files, minimal input data, stored results, and evaluation utilities.
• run_moo_es_parallel.sbatch: Example SLURM script for running MOEns on an HPC cluster.

Reproducibility Workflow - Minimal Example

The workflow consists of three parts. They can be executed independently if the required input data already exists.

Reproducing: Generating Base Models with Auto-Sklearn

Please refer to ./assembled_ask/README.md for details.

Reproducing: Running Post-Hoc Ensembling Methods

Please refer to ./assembled_ensembles/README.md for details.

Reproducing: Plots and Evaluations

Please refer to ./benchmark/evaluation/README.md for details.

Relevant Publications

If you use this repository, please cite the thesis and the original Q(D)O-ES codebase on which the repository builds.

@thesis{akay2026moens,
  title  = {Exploring Post-Hoc Multi-Objective Ensembling for Predictive Accuracy and Adversarial Robustness in AutoML},
  author = {Akay, Berkay},
  school = {RWTH Aachen University},
  year   = {2026},
  type   = {Bachelor's Thesis}
}

@inproceedings{
  purucker2023qdoes,
  title={Q(D)O-{ES}: Population-based Quality (Diversity) Optimisation for Post Hoc Ensemble Selection in Auto{ML}},
  author={Lennart Purucker and Lennart Schneider and Marie Anastacio and Joeran Beel and Bernd Bischl and Holger Hoos},
  booktitle={AutoML Conference 2023},
  year={2023},
  url={https://openreview.net/forum?id=zvV7hemQmtLl}
}