Streamline unit tests (review draft for microbiome/mia#748)

[raivo-otus]

Review draft inside the fork. This PR exists so the diff and the per-file commits can be reviewed on GitHub before the official pull request is opened against microbiome/mia:devel.

Why

Bioconductor devel (3.24) check of mia 1.21.7, 2026-09-15:

Node	Platform	CHECK	Elapsed
nebbiolo2	Linux x86_64	WARNINGS	2035 s, of which testthat.R 1404 s
lconway	macOS x86_64	TIMEOUT (40 min)	killed while running testthat.R
taishan	Linux aarch64	TIMEOUT (40 min)	killed while running testthat.R

The unit tests are ~70 % of the check time. Release 3.23 (1.20.0) already times out on the aarch64 node.

What changed

Only tests/testthat/ and one test fixture (inst/extdata/taxpasta_no_type.biom, used by a single test). No change under R/.

• Shared fixture tests/testthat/helper-data.R defines gp_small, a seeded 300-feature subset of GlobalPatterns (rows kept in tree order, rowTree pruned to 300 tips). Tests whose assertions are structural (dims, names, classes, warnings) or compare mia with an upstream package (vegan, ecodive, phyloseq, NMF, topicmodels, base R) run on it; tests with hard-coded expected values derived from the full object keep the full object.
• Data-size reductions: alpha-diversity rarefaction on family-level data with niter 3/5 instead of 10/20 over 19216 features; LDA on genus-level data; getCrossAssociation on the bundled HintikkaXOData instead of a network fetch of microbiomeDataSets::peerj32(); mockSCE(40, 300); IdTaxa on 20 sequences; a 40-observation taxpasta BIOM fixture.
• Fewer calls: the transformation helper no longer runs its storage-independent second half twice; rank loops run over two ranks instead of seven; byte-identical duplicate calls, self-comparisons and unused computations are removed; NMF multi-run is sequential (.options = "-p").
• Test-quality fixes (each in its own commit): four files had expect_equal(round(x, 7), round(ref), 7) with the closing parenthesis in the wrong place, so the reference was rounded to integers and the digit count became the tolerance; two expect_error calls passed because they referred to an undefined object; a loop over length(list) checked only the last element; getPERMANOVA was compared with adonis2 through a non-existent aov.tab element (NULL vs NULL); getTop and summarizeDominance were compared with themselves; random draws are now seeded; the internal richness helper disabled in 2024 passes again and is re-enabled (+22 expectations); optional packages (miaTime, ecodive) are guarded with skip_if_not_installed() before first use.

One commit per test file; each commit message states what was removed and what still covers the same behaviour.

Timing (local, 16-core Linux; the Bioc Linux node is roughly 2x slower)

Before: 708 s (all Suggests installed). After: **190 s (3.7x faster; 118 blocks, 0 failures, 1 skip for dada2). The tests part of a local R CMD check ([ FAIL 0 | WARN 5 | SKIP 1 | PASS 1376 ], same five pre-existing vegan warnings as before) took 202 s on a loaded machine; the Bioc Linux node needed 1404 s before. With the optional packages (NMF, topicmodels, mediation, reldist, topicdoc, philr, miaTime, dada2) absent, all their blocks skip and nothing errors (9 skips, 0 failures)**.

File	before (s)	after (s)	expectations before -> after
test-addAlpha.R	245	5.9	48 -> 48
test-5getCrossAssociation.R	59	14.8	71 -> 71
test-4splitOn.R	56	11.7	28 -> 36
test-5prevalence.R	55	28.6	111 -> 89
test-5addNMF.R	50	24.9	16 -> 16
test-5transformCounts.R	35	9.4	248 -> 182
test-2mergeSEs.R	32	15.0	89 -> 89
test-3agglomerate.R	28	13.8	53 -> 53
test-4IO.R	25	9.6	135 -> 130
test-2taxonomy.R	18	9.9	65 -> 65
test-importTaxpasta.R	16	4.5	15 -> 15
test-4splitByRanks.R	16	6.3	8 -> 8
test-2merge.R	15	5.7	62 -> 61
test-5addLDA.R	14	1.2	17 -> 17
test-0utilites.R	8	1.9	44 -> 44
test-6cca.R	6	4.7	30 -> 30
test-5dominantTaxa.R	4	4.2	8 -> 8
test-0diversity.R	4	1.3	37 -> 37
test-0divergence.R	3	1.8	25 -> 25
test-5dominance.R	2	2.1	26 -> 26
test-9cluster.R	2	2.0	20 -> 20
test-5Unifrac.R	2	0.8	14 -> 14
test-mediate.R	2	1.3	14 -> 14
test-addMDS.R	1	1.1	16 -> 16
test-getPERMANOVA.R	1	1.1	34 -> 34
test-5evenness.R	1	1.1	14 -> 14
test-5runDPCoA.R	1	0.2	16 -> 16
test-filterRPCAInput.R	1	1.0	34 -> 34
test-8subsample.R	1	0.9	10 -> 10
test-5getDissimilarity.R	1	0.6	8 -> 8
test-getAbundant.R	1	0.4	27 -> 27
test-8summaries.R	0	0.4	8 -> 10
test-5calculateOverlap.R	0	0.1	8 -> 8
test-utilization_functions.R	0	0.2	7 -> 7
test-7decontam.R	0	0.2	10 -> 10
test-5runNMDS.R	0	0.1	10 -> 10
test-getRPCA.R	0	0.1	44 -> 44
test-5richness.R	0	0.4	4 -> 26
total	708	189.6	1434 -> 1372

Coverage

covr::package_coverage() before/after on the same machine: line coverage 74.85 % -> 74.97 % (545 functions). Per function: no function lost a covered line, six gained (the re-enabled richness helper covers .calc_ace, .calc_chao1, .calc_hill; plus lines in mergeSEs, getCrossAssociation and .check_altExp_present).

Notes for reviewers

• test-addAlpha.R: two correlation sanity thresholds were lowered (camargo 0.7 -> 0.5, absolute 0.9 -> 0.8) because they are properties of the full data, not of the code; all other tolerances are unchanged. A random feature subset breaks the richness-estimator checks entirely, which is why family-level aggregation is used there.
• test-addMDS.R: while fixing a self-comparison it turned out that getMDS(..., node.label = <unnamed vector>) gives a different ordination than the default mapping, although getDissimilarity() agrees for both. The test now checks the named-vector case (which matches the default). The positional case may deserve a look in R/.
• The remaining cost is dominated by the fixed per-call overhead (~0.6-1 s) of agglomerateByRank(), splitOn()/unsplitOn() and agglomerateByRanks() regardless of data size; that is package code and out of scope here.
• Examples are unchanged and still take ~5 min on the Bioc Linux node (calculateDMN 65 s, addNMF 40 s, agglomerate-methods 27 s, hierarchy-tree 22 s); the same subset/nrun treatment would apply to them in a follow-up.

Supersedes the draft microbiome#749.

Profiling script (run against an installed build)

# Rscript profile_tests.R  -- per-test wall time, aggregated per file
suppressPackageStartupMessages({ library(testthat); library(mia) })
res <- testthat::test_dir("tests/testthat", package = "mia", load_package = "installed",
                          reporter = ListReporter$new(), stop_on_failure = FALSE)
df <- as.data.frame(res); df <- df[, vapply(df, function(x) !is.list(x), TRUE)]
byfile <- aggregate(cbind(real, nb, failed, skipped) ~ file, data = df, FUN = sum)
print(byfile[order(-byfile$real), ], row.names = FALSE)

Fixture derivation (inst/extdata/taxpasta_no_type.biom)

# Build a small copy of inst/extdata/taxpasta_no_type.biom (HDF5 BIOM 2.1 with
# an empty 'type' attribute, as written by taxpasta) keeping 40 observations
# with counts, chosen so that every taxonomy column is populated. Mirrors the
# layout of the source file. Usage:
#   Rscript this_script.R <old.biom> <new.biom> 40
suppressPackageStartupMessages(library(rhdf5))
args <- commandArgs(trailingOnly = TRUE); src <- args[1]; dst <- args[2]; n_keep <- as.integer(args[3])
ids <- as.vector(h5read(src, "observation/ids"))
tax <- h5read(src, "observation/metadata/taxonomy")           # dims as read by rhdf5
ranks <- as.vector(h5read(src, "observation/group-metadata/ranks"))
sids <- as.vector(h5read(src, "sample/ids"))
o_data <- as.vector(h5read(src, "observation/matrix/data"))
o_indptr <- as.vector(h5read(src, "observation/matrix/indptr"))
cat("taxonomy dims as read:", dim(tax), " ids:", length(ids), "\n")
# observations with at least one count (one sample -> at most one entry each);
# choose rows so that every taxonomy column is populated at least once, then
# fill up with the rows that have the deepest lineages
counts <- diff(o_indptr)
tax_by_row <- if (ncol(tax) == length(ids)) t(tax) else tax   # rows = observations
nonempty <- tax_by_row != ""
cand <- which(counts > 0)
keep <- integer(0)
for (j in seq_len(ncol(nonempty))) {
    hit <- cand[nonempty[cand, j]]
    hit <- setdiff(hit, keep)
    if (length(hit)) keep <- c(keep, hit[which.max(rowSums(nonempty[hit, , drop = FALSE]))])
}
depth <- rowSums(nonempty)
extra <- setdiff(cand[order(-depth[cand])], keep)
keep <- sort(c(keep, extra[seq_len(max(0, n_keep - length(keep)))]))
cat("kept", length(keep), "observations; populated columns:", sum(colSums(nonempty[keep, , drop = FALSE]) > 0), "of", ncol(nonempty), "\n")
vals <- o_data[o_indptr[keep] + 1]
# taxonomy rows: rhdf5 returns the (nobs x nranks) HDF5 dataset transposed
tax_keep <- if (ncol(tax) == length(ids)) tax[, keep, drop = FALSE] else t(tax[keep, , drop = FALSE])
attrs <- h5readAttributes(src, "/")
if (file.exists(dst)) file.remove(dst)
h5createFile(dst)
for (g in c("observation", "observation/matrix", "observation/metadata", "observation/group-metadata",
            "sample", "sample/matrix", "sample/group-metadata")) h5createGroup(dst, g)
n <- length(keep)
h5write(ids[keep], dst, "observation/ids")
h5write(tax_keep, dst, "observation/metadata/taxonomy")
h5write(ranks, dst, "observation/group-metadata/ranks")
h5write(vals, dst, "observation/matrix/data")
h5write(rep(0L, n), dst, "observation/matrix/indices")
h5write(as.integer(0:n), dst, "observation/matrix/indptr")
h5write(sids, dst, "sample/ids")
h5write(vals, dst, "sample/matrix/data")
h5write(as.integer(seq_len(n) - 1), dst, "sample/matrix/indices")
h5write(as.integer(c(0, n)), dst, "sample/matrix/indptr")
fid <- H5Fopen(dst)
for (a in c("creation-date", "format-url", "format-version", "generated-by", "id", "type")) h5writeAttribute(attrs[[a]], fid, a)
h5writeAttribute(as.numeric(n), fid, "nnz")
h5writeAttribute(as.numeric(c(n, 1)), fid, "shape")
H5Fclose(fid)
h5closeAll()
cat("written", dst, "with", n, "observations;", file.size(dst), "bytes\n")
print(h5ls(dst)[, c("group", "name", "dim")])

🤖 Generated with Claude Code