When you build an AI enrichment pipeline, the default temptation is to automate the output. Suggestions go in, classifications come out, humans correct errors when they surface. It’s faster to build and faster to run.
.
It’s also wrong — at least for a system where the classification decides what a learner can find.
.
What the pipeline actually does
The AI Enrichment Tools pipeline in WhereWeLearn analyses lesson content and produces three kinds of suggestion: relevant occupational role tags, UDC subject code and IFQ (International Framework of Qualifications) level pair that make a lesson discoverable — and a difficulty rating within its subject domain.
.
None of it goes live automatically. Every suggestion lands in a review queue first. An educator opens the queue, and for each suggestion approves it, edits it, or rejects it. Only after that step does the classification apply to the lesson. The wiki page that documents the pipeline states the position plainly: “AI suggestions are starting points, not final answers.”
.
The same pattern held at larger scale. When 701 IWTB (I Want To Be) career roles were run through the enrichment pipeline — hashtags, synonyms, classification codes generated in two API passes — every one of the 701 outputs went through the same queue before it was visible to a learner. The AI produced 701 drafts in hours. A human still looked at every one.
.
Why not auto-apply
The alternative — apply suggestions automatically, fix mistakes when someone notices — is available. It just costs more than it looks like it costs.
.
**Error invisibility.** An auto-applied classification error doesn’t announce itself. A lesson about programming filed under history sits there, misplaced, until a learner who wanted a programming lesson never finds it and a learner who wanted history stumbles onto it by mistake. Nobody reports an error they never see happen. The feedback loop that would catch it runs through learner confusion, not a log file, and it can take months to close.
.
**Educator disengagement.** If the AI classifies everything and it’s usually fine, educators stop checking. Not from negligence — from the ordinary way attention allocates itself away from things that seem to work. The review habit atrophies exactly when it’s still needed, because “usually fine” is not the same as “always fine,” and nobody is watching for the difference anymore.
.
**The asymmetry.** Auto-application optimises for throughput. The cost of getting it wrong — a learner who can’t find what they need, a recommendation that points at the wrong content — lands on someone else, later, in a way that’s hard to trace back to the original decision to skip review. Whoever chose to skip the review is not the one who pays for the mistake.
.
What the queue buys back
.
Putting a human in front of every suggestion is slower. It also produces three things auto-application can’t.
.
A domain expert has looked at the classification, not just the confidence score attached to it. Rejections are logged rather than silently overwritten, which means the system has a running record of where the AI is systematically wrong — a signal that disappears entirely if nothing is ever rejected in the open. And educators know their curriculum isn’t being rewritten without their knowledge, which matters as a professional and, in some contexts, a regulatory concern, not just a feeling.
.
In practice it looks like this. A lesson goes through the pipeline and produces three suggestions: a role tag of “Software Developer,” an EKOS subject code of `004.42` (programming), and a difficulty rating of Level 3 of 5. The educator approves the role tag and the subject code — both correct — and rejects the difficulty rating, moving it to Level 2. That rejection is logged. Enough rejections clustered on introductory-level difficulty ratings would flag a pattern worth investigating in the model, not just in that one lesson.
.
The decision underneath it
.
“Human oversight of AI output” reads as an obvious best practice in AI safety writing. In an actual production pipeline it is a choice, made against real pressure — speed-to-publish favours skipping the queue, and nothing in the architecture forces you to keep it.
.
The decision here was made explicitly, for a specific reason: classification is what learners depend on to find content at all. When discovery itself is downstream of a suggestion, accuracy has to outrank throughput. The friction isn’t a cost the pipeline pays reluctantly. The friction is the quality control.
.
Most AI-augmented workflows automate first and ask questions when something breaks. The cost of that ordering is paid later, quietly, in wrong classifications nobody traced, in review habits that eroded because nothing seemed to need them, in a feedback loop that never closes because no one is tracking what got rejected.
.
If a pipeline is producing suggestions that matter, the question worth asking isn’t how to make it faster. It’s what error rate is acceptable, and who ends up bearing the cost of the errors it lets through.
