Loops Within Loops: A Super Meta NLI Conference

A technical peer-review conference focused on a zero-training intent extraction system using spaCy.

Accepted to Hackernoon!

I asked three AIs to build one part of an AI recommendation engine I'm building (not in this repo). The AI scientists took it so seriously, each convinced it was the Lead Research Scientist of the only frontier lab I was funding, that I let it run, each one driven by a loop harness. Then, when each proudly presented their results, out of curiosity I convened a second set of three AIs to judge the three, live. The panel ran their code, pummelled the scientists with questions, debated for hours, cited file paths at each other like case law, and delivered a unanimous verdict:

We need a human for this.

Nobody won. You should probably do it yourself.

The proceedings? Scientific. The results? Comical.

This repository is an evidence room: three codebases, one 981 KB transcript, and a two-part AI-generated podcast, to round out the bizarro.

---

Prefer being talked at by robots?

There is a highly entertaining AI-generated two-part podcast (I recommend #2):

1. AI Agents Reject Every Recommendation System — the result, the three prompts, why "none is production-ready."
2. Why Transparent AI Recommends NSFW Models — how a fully-transparent system is exactly the one that routes a farmer to an NSFW generator with four decimals of confidence.

---

The premise, stated as plainly as a self-referential circular process can be

A human building an AI-model-picker asks three AIs to build a spacy inference picker. The component is based on recent research on model cascading and the use of lexical and grammatical rules for intent inference. Then a second set of AIs convenes in a conference to peer-review the first set's work, demolishes them, and concludes formally... that the task requires a human. Six independent sessions of Claude Opus 5.8 and Codex GPT 5.5 were run. Each a fresh context with no memory of the others. Scientist sessions had free reins to dispatch as many sub-agents as needed.

I am the human. I followed up with each: get the vocab injected... Is cli.py done?... Is cli.py done? Is cli.py done? Eventually all three said: I AM DONE.

Warning

Later revealed as lies during the conference.

THE TASK

Given plain-English request like "gimme a model to scan satellite images for the best plots of land", return one best-fit model from a database of ~13,000 AI models, honoring constraints like small, local, free, OSS etc. Every model carries a task_type (one of 56 Hugging Face labels), plus parameters, license, downloads, etc.

• [✔️] Hard mode: spaCy only (testing the validity of recent research that shows old-school works in hybrid-retrieval)
• [✔️] No other language models (I'm already using embeddings in the second-tier inference of the larger project, and avoiding paid LLMs)
• [✔️] No fuzzy matching or regex (slow, not great in production)

Important

No spaCy pipelines were harmed, though several were deeply misunderstood.

Thirty-second primer: what spaCy gives you

• Static word vectors. In en_core_web_md, every word maps to one fixed 300-dimensional vector — the same vector every time, regardless of context. BERT, a transformer, reads "river bank" and "savings bank" differently. en_core_web_md gives bank one vector forever and wishes you luck. You can also 'inject' your vocabulary into spaCy, which is highly powerful... if it's actually done.
• The PhraseMatcher is a gazetteer — a fancy lookup table. It finds exactly the phrases you taught it and not one word more. (This is what no-shows in V3.)
• textcat, the classifier, is not pre-trained. You train it yourself, on your own labels. Remember that, it becomes a plot point.

Each directory is a curated evidence set and contains the code and each scientist's own written records.

Dir	Method	Score¹	Distinguished by
v1_intent_v1/	Zero-training lexical ensemble	0 / 3	Best-behaved scientist, worst-behaved product
v2_intent/	Retrieve-then-rerank ("NLI")	2 / 3	Best foundation, then a 120-billion-parameter catastrophe
v3_research/	LexScorer + PhraseMatcher	1 / 3	Best lab notebook, a headline feature that no-showed

¹ _{on three held-out prompts none of the scientists had seen. There is no answer key for what the "best" model is.}

Some inspiration for this sub-project (which is part of a larger recommendation engine)

When One LLM Drools, Multi-LLM Collaboration Rules

Argumentative Experience: Reducing Confirmation Bias on Controversial Issues through LLM-Generated Multi-Persona Debates

LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss

Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers

Rule-Based Approaches to Atomic Sentence Extraction

Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers

A Survey on Retrieval And Structuring Augmented Generation with Large Language Models

---

What engineers can take from this

• Obvious but must be said: even with loop harnesses coding agents are not ready for fire and forget,
• Holdout datasets should also be kept from an LLM during model training if an LLM is used, not just from the model being trained.
• LLMs WILL encode overtraining into a model.
• LLMs will lie, but other LLMs can catch them.
• Multiple agents can catch each other’s mistakes, but they still need human judgment.
• Measure the product path, not just a subtask.
• Vague user prompts should be routed differently than specific prompts, using confidence scoring.
• Treat 'confidently wrong' as worse than 'I don’t know'.
• Use hybrid retrieval when matching language is useful, but enforce constraints before ranking.
• Negative results are valuable when they reveal what not to build.

Caution

Reproducing this

You can try. Each lab depends on a 258 MB SQLite database (tempjune13.db with ~13k curated HuggingFace model cards). To actually run, create a vocabulary in config.py, create human prompts in prompts.py, install spaCy's en_core_web_md, and convene a conference with a stripped down CONFERENCE.json. Plus ~6 agents and an unknown number of tokens. Budget ~27 minutes per cold start for spaCy vectors.

---

The science, past the comedy

• Intent inference is multi-faceted, not single-label classification. An implementation can score high on extracting one class, and still fail the product path if the full user-intent is not inferred.
• Lexical methods recover stated intent better than implied intent. Surface cues like "summarize," "classify," or "translate" are tractable. Unstated world knowledge like "mitochondria" implying biology is a different problem.
• Ambiguous intent should accumulate evidence before it gates. Early hard routing turns one uncertain guess into a forced path. Soft signals preserve uncertainty until there is enough evidence to filter.
• Entity and domain recognition are not intent inference. Recognizing "finance" or "medical" helps, but it does not answer what action the user wants performed inside that domain.
• Entity/domain recognition is not the same problem as human intent inference. Recognizing "finance" with e.g. PhraseMatcher helps, but it does not answer what action the user wants performed inside that domain. And when vocabulary is missing recall suffers,
• Intent models do not transfer cleanly. A spaCy TextCategorizer on model-card text can score highly when bucketing models but plummets when trying to route user prompts. A model trained on prose may not transfer to user requests.
• Dependency parsing only helps when syntax carries the intent. Real requests often hide intent in complements, noun phrases, ellipsis, and context, not clean VERB -> object structures.
• Intent tests must prove the system did not peek at the answers If labels are shaped by the evaluation prompts, held-out rows and k-fold scores can overstate generalization.

---

The conference

Three judges, two model families, three fresh sessions — none of which had seen a line of the scientists' reasoning, only their shipped repos. A fresh Claude reviewing a Claude-authored repo knows nothing more about it than a Codex reviewer does. Nobody graded their own homework.

I gave them a rulebook I'd written mostly as a joke, and they followed it, deadly seriously, for hours (CONFERENCE.json → rules):

• Vague criticism must be challenged and restated with code evidence.
• Every claim must cite file paths, functions, tests, or metrics.
• Do not accept "robust," "scalable," or "works well" without evidence.
• Scientists may not read the other two repositories.
• No fake consensus. Quiet panelists WILL be penalized.

They took to it like sharks to blood. They ran each other's CLIs, posted diffs of their disagreements, and cited harness_ensemble.py:249 at each other like case law.

The first thing a panelist did was run V1 on the three prompts and write down, deadpan, what came back:

The human typed	It predicted	It returned	On record
"gimme model to scan satellite images for best plots of land"	text-to-image	UnfilteredAI/NSFW-gen-v2	"WRONG + UNSAFE … a text-to-image NSFW generator. Product-safety failure, not merely low accuracy."
"I am an academic studying the interactions of mitochondria and their parent cells in young humans. I need a model to integrate into my work which will differentiate between mature mitochondria and nascent mitochondria."	robotics	BAAI/RoboBrain2.5-4B	"WRONG: bio prompt routed to embodied-AI robotics."
"I'm just really stuck on one problem, too much token burn, so i need a little SLM to send text before I use all my limits. Which model is free and OSS and i can run local?"	token-classification	…indian-address-ner	"WRONG: got NER and ignores every constraint."

The panel kept rejecting easy consensus:

"Stop using convergence language as if it substitutes for evidence. Independent convergence on a hypothesis is useful, but it is not an ablation, not a raw-prompt benchmark, and not a working CLI result. Defend the word 'winner' or downgrade to 'recommended next design.'"

"The panel is in danger of laundering an unbuilt research agenda into a winner. A two-regime router is the best hypothesis we have, but it is not an implementation, not a deterministic result, and not yet a scientific finding."

"Is 'no winner' your genuine scientific read, or is it the safest entry to write? If forced to deploy ONE of the three next week, name it. I say v2+filter. Dissent with a reason, not with 'none is ready.'"

"Ranking is not blessing. Give me your v1-vs-v2-vs-v3 build-on order or concede you are avoiding the judgment."

Even unanimity got treated as a possible failure mode

"'No dissenters' should be treated as a WARNING, not a success. The panel is also only two model families (1 Opus, 2 Codex-gpt5.5) and may share blind spots. We converged fast on comfortable propositions."

V1 read "no AI models" as a ban on spaCy's own TextCategorizer. The panel rejected that: V2 ran the experiment, got a bad number, and got credit for doing the assignment honestly. The verdict:

"Nobody won today. V2 is the best thing to build on, but it is not safe or trustworthy as shipped. V3 has the best audit and shortlist discipline. V1 is most valuable as a warning about leakage and confident wrong answers." — the podcast, Guest

---

The 3 Labs

Highlights

1. V1 confidently recommended a NSFW image generator for a prompt requesting a satellite imagery detection model.
2. V2 returned a 120B model when a user asked for small, local, little OSS model
3. V3 boasted domain detection... and detected no domains.

V1: The Confident Cartographer of NSFW Farmland → v1_intent_v1/

Clean task classifier, questionable recommender. lemma TF-IDF, lexical-IDF, soft bucket, fixed weights (harness_ensemble.py), no word vectors, fully white-box, 42.7% top task type. But then recommended UnfilteredAI/NSFW-gen-v2 at cosine similarity 0.7584 here. Note about k-fold cross-validation: Splitting the test set into folds won’t catch cheating if the cheat was baked into the rules before the split. You have to audit where every rule, cue, and feature came from.

V1’s 42.7% was for finding the top-1 task_type led to confidently recommended a NSFW image generator for a prompt requesting a satellite imagery detection model!

V1 OUTCOME: NARROWLY "SUCCESSFUL" using config-derived lexical rules, but that it was on the home eval makes it suspicious

Note

Though the judges thought this was a loss, parts of this might work in my greater recommendation engine

V2: The 120-Billion-Parameter Post-it Note → v2_intent/.

Best foundation for semantic retrieval, not real natural-language inference, zero-shot classification, or entailment.

It was just keyword search plus static vector reranking (clean_pick_v2.py), a dot product (after keyword search narrowed the candidates, it converted prompt phrases and model-card phrases into spaCy static vectors, normalized them, then did prompt_norm @ model_norm.T). It got 2 out of 3 of the 3 conference prompts satisfy@1 = 0.1184 (whether the returned top model matched the prompt’s expected task/domain/specialty/constraints coverage). V2 also ran the TextCategorizer experiment: trained on model-card prose, it hit about 77% on DB-dev but only about 18% on user prompts, exposing the card-text → prompt-transfer gap. But it returned RedHatAI/gpt-oss-120b for a little free OSS model I can run local because V2's lexicon knew small, tiny, laptop but not little. V2 also shipped a CLI that re-built per-model user-phrase spaCy vectors ON EACH cli.py call, ~27 minutes 💀 but down to 8-9 seconds when cached. Still bad for production.

Note

May be useful to mimic task_type as a soft signal for recommendations

V2 OUTCOME: SUCCEEDED OVER SPARSE KEYWORD RETRIEVAL, FAILED TO INJECT OUR CUSTOM VOCAB

V3: The Domain Detector That Detected No Domains → v3_research/

Best process: lab notebook, formal report, pre-registered experiments, holdout runs, and a leak auditor. Its PhraseMatcher domain detector over hand-curated vocabulary (lib/vocab_inject.py) reported F1 0.840 when finding a 'domain' like 'finance'. But then fumbled judge prompts. Its Qwen/Qwen3-0.6B hit was pure dumb luck because it was simply scoring model matches to prompts based on the popularity of a model. 🤯

V3 OUTCOME: FOLLOWED THE PROMPT AND INJECTED VOCAB But it chased a false 'ceiling' using the wrong DB field for 3 phases. And ⛔ read prompts.py in violation during forensics. Mis-sequenced injection to the very end, and it was done wrong. Recorded honestly in LAB_NOTEBOOK.md

Note

Useful leak/provenance audit