diff --git a/inst/WORDLIST.txt b/inst/WORDLIST.txt index d1442430..3f63f202 100644 --- a/inst/WORDLIST.txt +++ b/inst/WORDLIST.txt @@ -195,6 +195,7 @@ Bearham Becca behaviour ben +benchmarking Berel BILI BILIBL @@ -345,9 +346,9 @@ dat datacutr dataname datanames -dataset's datasetjson DatasetMeta +dataset's datatable dataview dataviewer @@ -584,7 +585,6 @@ htmlwidgets httpuv httr hu -ℹ Iaccarino IBMRounding ICH @@ -617,6 +617,7 @@ iscr ISCR ITT ITTFL +ℹ Jagadish Janssen Javascript @@ -629,8 +630,8 @@ jquerylib js json JSON -JSON's jsonlite +JSON's jygn kable Kalappurakal @@ -651,7 +652,6 @@ kronecker Krouse Kumar Kumari -L'Ecuyer labsbl lang lapack @@ -666,6 +666,7 @@ lbt LBTESTCD LBTESTCDB learnings +L'Ecuyer Leena Lepore Lewandowski @@ -687,8 +688,8 @@ locf lockfile logr logrx -logrx's logrxaddin +logrx's LSTALVDT lsting LTS @@ -941,6 +942,7 @@ readRDS readxl reb registrationData +reimplemented relevel RELREC renderUI @@ -992,8 +994,8 @@ RSPM rstudio rStudio RStudio -RStudio's rstudioapi +RStudio's rtable rtables rtf @@ -1087,6 +1089,7 @@ SUBJTYP SUBJTYPC subline submisisons +substr subteam Suekuer Sumesh @@ -1309,4 +1312,4 @@ zlib zMachine Żyła zzz -β +β \ No newline at end of file diff --git a/posts/2026-06-14_admiral_in_age_of_ai/admiral.png b/posts/2026-06-14_admiral_in_age_of_ai/admiral.png new file mode 100644 index 00000000..eacdc9ba Binary files /dev/null and b/posts/2026-06-14_admiral_in_age_of_ai/admiral.png differ diff --git a/posts/2026-06-14_admiral_in_age_of_ai/admiral_in_age_of_ai.qmd b/posts/2026-06-14_admiral_in_age_of_ai/admiral_in_age_of_ai.qmd new file mode 100644 index 00000000..f4634e90 --- /dev/null +++ b/posts/2026-06-14_admiral_in_age_of_ai/admiral_in_age_of_ai.qmd @@ -0,0 +1,68 @@ +--- +title: "Why we still need {admiral} in an age of AI" +author: +- name: Jeff Dickinson +description: "LLMs can write R code, but can they write reliable and readable ADaM code? Benchmark data from the R Consortium pharma-skills initiative shows that without structured {admiral} skill guidance, AI agents produce inconsistent error-prone derivations. Here's what the data shows and why the pharmaverse package ecosystem matters more than ever." +date: "2026-06-14" +categories: [Technical, ADaM, Community] +image: admiral.png +--- + + + +```{r setup, include=FALSE} +long_slug <- "2026-06-14_admiral_in_age_of_ai" +library(link) +link::auto() +``` + + + +There is a version of the AI-in-pharma story that goes like this: LLMs are trained on vast amounts of R code, so they can write ADaM programs on demand. Packages like {admiral} become optional — a style preference rather than a requirement. Just describe what you need and let the model figure it out. + +The benchmark data from [pharma-skills](https://github.com/RConsortium/pharma-skills) tells a different story. + +## What an Unskilled Agent Actually Does + +When an AI coding agent is asked to derive an ADAE dataset without access to {admiral} skill guidance, it does not reach for `derive_var_trtemfl()` or `derive_vars_merged()` with the correct parameters. Across multiple benchmark runs, unskilled agents fell into two consistent failure modes: either generating synthetic data rather than using pharmaverse reference datasets, or writing bespoke `dplyr` pipelines that reimplemented logic {admiral} already provides — incorrectly. + +One example from BDS benchmarking is particularly telling. Without skill guidance, agents consistently used `derive_vars_merged()` where `derive_vars_merged_lookup()` was required for parameter code assignment. Both functions exist in {admiral}. Both execute without error. But `derive_vars_merged()` drops unmatched records silently, producing a dataset with the wrong row count. No warning. No crash. Just wrong output that passes a casual review. + +This is not a model quality problem. It is a knowledge problem. The model does not know what the pharmaverse community knows. + +## The Package as Specification + +{admiral} is more than a collection of R functions. It is a community-maintained encoding of CDISC ADaM logic — accumulated through years of collaboration across sponsors, CROs, and regulators, tested against real submissions, and versioned for traceability. When a programmer calls `derive_vars_dtm()` with the correct imputation flags, they are not just writing R code. They are implementing a specification that has been reviewed, validated, and documented. + +An LLM trained on general R code does not reliably inherit that specification. It has seen {admiral} in its training data, but not with the depth or precision needed to apply it correctly across the full range of ADaM derivation scenarios. The unskilled agent that wrote a custom `parse_dtc_datetime()` function using `substr()` and `as.POSIXct()` — rather than calling `derive_vars_dtm()` — was not being lazy. It was doing its best with what it knew. Its best was not good enough, and the errors it introduced were in the edge cases that matter most in a clinical submission. + +## What the Skill Does + +The {admiral} skills in pharma-skills do not replace {admiral}. They connect the AI agent to it. A skill provides curated, domain-aware guidance: which functions to use for which derivations, how to structure the program for QC readability, which variables require special handling, and what assertions to include. The skill is the bridge between a capable general-purpose model and the specific, validated logic the pharmaverse community has built. + +The benchmark results reflect this directly. Across ADSL, ADAE, ADVS, and ADLB: + +- **With skill: 88–100%** pass rates across domains +- **Without skill: 17–59%** pass rates, with high variance + +That variance in the unskilled condition matters as much as the mean. Inconsistent output is not a defensible process in a GxP context. A skill-guided agent produces consistent, traceable, {admiral}-anchored code. An unskilled agent produces something different every time. + +## The Accountability Anchor + +There is a regulatory dimension here that goes beyond code quality. A clinical submission needs to trace its derivations to validated, versioned, documented methods. A bespoke LLM-generated pipeline — however functional — has no such anchor. {admiral} provides it. When a submission uses `derive_var_trtemfl()` from a pinned version of {admiral}, the derivation logic is documented, community-reviewed, and reproducible. The AI is most useful when it is writing code that inherits those properties, not when it is improvising around them. + +This is why the pharma-skills project frames skills not as prompt templates, but as domain knowledge artifacts. The goal is not to make AI write more R code. It is to make AI write {admiral} code — correctly, consistently, and in a form that a human reviewer can audit and a regulatory submission can defend. + +{admiral} was built for exactly this moment. The community just needs to make sure AI knows how to use it. + + + +```{r, echo=FALSE} +source("appendix.R") +insert_appendix( + repo_spec = "pharmaverse/blog", + name = long_slug, + # file_name should be the name of your file + file_name = list.files() %>% stringr::str_subset(".qmd") %>% first() +) +``` diff --git a/posts/2026-06-14_admiral_in_age_of_ai/appendix.R b/posts/2026-06-14_admiral_in_age_of_ai/appendix.R new file mode 100644 index 00000000..83c59e21 --- /dev/null +++ b/posts/2026-06-14_admiral_in_age_of_ai/appendix.R @@ -0,0 +1,73 @@ +suppressMessages(library(dplyr)) +# markdown helpers -------------------------------------------------------- + +markdown_appendix <- function(name, content) { + paste(paste("##", name, "{.appendix}"), " ", content, sep = "\n") +} +markdown_link <- function(text, path) { + paste0("[", text, "](", path, ")") +} + + + +# worker functions -------------------------------------------------------- + +insert_source <- function(repo_spec, name, + collection = "posts", + branch = "main", + host = "https://github.com", + text = "Source", + file_name) { + path <- paste( + host, + repo_spec, + "tree", + branch, + collection, + name, + file_name, + sep = "/" + ) + return(markdown_link(text, path)) +} + +insert_timestamp <- function(tzone = Sys.timezone()) { + time <- lubridate::now(tzone = tzone) + stamp <- as.character(time, tz = tzone, usetz = TRUE) + return(stamp) +} + +insert_lockfile <- function(repo_spec, name, + collection = "posts", + branch = "main", + host = "https://github.com", + text = "Session info") { + path <- "https://pharmaverse.github.io/blog/session_info.html" + + return(markdown_link(text, path)) +} + + + +# top level function ------------------------------------------------------ + +insert_appendix <- function(repo_spec, name, collection = "posts", file_name) { + appendices <- paste( + markdown_appendix( + name = "Last updated", + content = insert_timestamp() + ), + " ", + markdown_appendix( + name = "Details", + content = paste( + insert_source(repo_spec, name, collection, file_name = file_name), + # get renv information, + insert_lockfile(repo_spec, name, collection), + sep = ", " + ) + ), + sep = "\n" + ) + knitr::asis_output(appendices) +}