Pattern write-up · LLM pipelines

Build vs buy for AI, decided by arithmetic

built with Bedrock · Lambda · Step Functions I wrote about this — the evals post

Content operations run on repetitive per-item work: every image needs alt text, every product needs enriched metadata, every new market needs translation. Vendors charge per item for all of it, and that pricing makes sense at small volume. Generative models can do this work now, if you can trust the output. This is the pattern I use for deciding when to bring that work in-house, and for shipping it once you do.

The pattern

Take content capabilities like alt-text generation, metadata enrichment, and translation, and build them as one pipeline on AWS Bedrock, Lambda, and Step Functions instead of separate bespoke tools. They share ingestion, orchestration, and the evaluation harness; prompts, models, and review flows differ per capability.

alt text metadata translation one pipeline Bedrock · Lambda eval gate production fails the gate → doesn't ship
Three capabilities on one pipeline pattern. Everything passes the eval gate before production.

Deciding what to build

Build vs buy, decided per capability. Only build where volume times per-item vendor pricing exceeds the cost of owning a pipeline. Run that arithmetic per capability rather than for "AI" as a whole. It will rule some capabilities out.

Evals as the launch gate. A capability ships only after clearing its evaluation suite — golden sets scored against editorial standards, with human QA sampling in production. Looking good in a demo doesn't count. This is the discipline I've written about: if the golden-set scores aren't there, it doesn't ship.

Right-size the model per task. Different capabilities have different quality floors and cost ceilings. Model choice per capability, rather than one flagship model everywhere, is a large part of why the math works.

Why it works

Where I've applied this, editorial teams end up reviewing model output rather than producing every item by hand. Each new capability starts from a working pipeline and evaluation harness. The pattern outlasts any one capability.

← Back to writing