How to scope an AI integration project (without burning six months)

Most AI projects don't fail in production. They fail in scoping.

By the time a team is debugging hallucinations in production, the expensive mistakes were already made months earlier: the wrong problem was chosen, success was never defined, cost was never modeled, and there was no way to tell whether the feature was getting better or worse. The model became the focus when it should have been the smallest part of the conversation.

This article is the scoping framework we actually use when a client says "we want to add AI." It is deliberately unglamorous. The goal is to leave a kickoff with a written definition of done, a cost model, an eval plan, and a staged delivery schedule — before anyone writes a prompt.

Start with the decision, not the model

The first question we ask is never "which model should we use?" It is "what decision or action does this feature change?"

LLM features earn their keep in one of a few ways:

They compress time. A task that took a human ten minutes now takes thirty seconds with review.
They unlock scale. Something that was economically impossible to do by hand becomes cheap enough to do for every record.
They improve an experience. Search gets better, support gets faster, onboarding gets smoother.

If you can't articulate the decision the feature changes, you don't have a project yet — you have a demo idea. "Summarize support tickets" is a demo. "Cut first-response time on tier-1 tickets by routing and drafting replies an agent approves in under 20 seconds" is a project. The second framing tells you what to measure, who the user is, and where the human stays in the loop.

Frame the problem as a measurable target

Every AI feature we ship has a written quality target before implementation starts. Non-deterministic systems are unmanageable without one. The target has three parts:

A quality metric. Accuracy, F1, groundedness, win-rate against a baseline, or a task-specific rubric scored by a stronger model or a human. Pick one primary metric and at most two guardrail metrics.
A threshold to ship. "85% of summaries rated 4/5 or higher by a reviewer" is shippable. "It feels good" is not.
A baseline. What is the current state — a human doing it manually, a regex, a keyword search? You compare against that, not against perfection.

This sounds heavy for a "simple" feature. It isn't. Writing the target forces the hard conversations early, when they're cheap. We have killed projects in the scoping phase because the only acceptable quality bar was one no current model could clear at an acceptable cost. That is a good outcome — it saved a quarter.

Build the eval harness before the feature

If there is one practice that separates AI features you can operate from features you can only hope work, it is this: build the evaluation harness before you build the feature.

An eval harness is a dataset of representative inputs plus a way to score outputs automatically. It is the test suite for a probabilistic system. Without it you are flying blind — every prompt tweak is a guess, every model upgrade is a gamble, and every regression ships silently to users.

A minimal harness has three components:

A golden dataset. 50–200 real, representative examples with known-good outcomes or rubrics. Pull these from production data, not your imagination. Edge cases belong here on purpose.
A scorer. Exact match where possible, an LLM-as-judge with a tight rubric where not, and human spot-checks to validate the judge.
A runner wired into CI. Every prompt change, model swap, and dependency bump runs the suite and reports the score delta.

Once this exists, iteration becomes engineering instead of vibes. You can answer "did that change help?" with a number. You can gate deploys on it. You can swap models the day a cheaper one launches and know within minutes whether quality held.

Model the cost before you commit

AI features have a unit cost that scales with usage, and it is a real line item — sometimes the dominant one. Model it during scoping, not after the first invoice.

The model is simple in shape:

cost per request = (input tokens × input price) + (output tokens × output price)
monthly cost     = cost per request × requests per month

The traps live in the details:

Retrieval and context stuffing inflate input tokens. A RAG feature that injects 8,000 tokens of context per call costs far more than the prompt suggests.
Agentic and multi-step flows multiply calls. A "single" feature that plans, calls tools, and reflects might be five model calls, not one.
Retries and fallbacks count. Budget for them.

Run the math at expected and peak volume. If the cost per successful outcome is higher than the value it creates, the architecture has to change — a smaller model for the easy cases, caching, batching, or a cheaper retrieval strategy. Better to learn that on a spreadsheet than on a credit card.

Design the failure modes explicitly

A demo handles the happy path. A product handles the bad day. During scoping we enumerate how the feature fails and what happens when it does:

The model is wrong but confident. What is the blast radius? Is there a human checkpoint? Can the user tell?
The model is slow. What is the latency budget, and what happens when it's exceeded — a spinner, a timeout, a degraded fallback?
The provider is down or rate-limiting. Is there a secondary provider, a queue, a graceful "try again later"?
The output is unsafe or off-brand. What filtering, validation, or schema enforcement sits between the model and the user?

The answers shape the architecture. A feature where a wrong answer is mildly annoying gets a different design than one where a wrong answer is a compliance incident. Deciding the tolerance during scoping is what keeps the build honest.

Choose the smallest architecture that hits the target

There is a natural ladder of complexity, and the discipline is to climb only as high as the quality target demands:

A single well-engineered prompt against a capable model. Try this first. It is astonishing how often it is enough.
Prompt plus retrieval (RAG) when the task needs facts the model doesn't have. Now you own chunking, embeddings, freshness, and retrieval quality — a real subsystem.
Tool use and structured output when the model needs to act or return machine-readable data. Schema validation becomes mandatory.
Multi-step or agentic flows when the task genuinely can't be done in one shot. This is the most powerful and the most expensive, slowest, and hardest to evaluate. Reserve it for when the simpler rungs provably fall short.

Every rung up the ladder adds cost, latency, and failure surface. "Boring and sufficient" beats "sophisticated and fragile" every time someone has to operate it at 2 a.m.

Stage the delivery

We deliver AI features in stages with explicit go/no-go gates, because the biggest risk — "will it actually work well enough?" — should be retired as early and as cheaply as possible.

Stage 0 — Spike (days). Build the eval dataset and run the simplest prompt against it. This either proves feasibility or kills the project before real money is spent.
Stage 1 — Internal beta. Ship behind a flag to internal users with full logging, cost tracking, and the human always in the loop. Measure against the target.
Stage 2 — Limited rollout. A small slice of real users. Watch cost, latency, and quality dashboards. Tune.
Stage 3 — General availability. Full rollout with alerting on quality regressions, cost spikes, and latency, plus a runbook for drift and incidents.

Each gate is a decision point with data behind it. Sometimes the data says stop, and stopping is the correct, money-saving call.

What "done" looks like

A well-scoped AI integration ends with more than a working feature. It ends with an eval suite that catches regressions before users do, versioned prompts and models you can roll back, cost and latency dashboards with alerts, and a runbook for the inevitable bad day. The feature is operable, not just impressive.

The model really is about 20% of the work. The other 80% — framing, evals, cost control, failure handling, and staged delivery — is ordinary engineering discipline applied to an unusually slippery component. Scope for that, and the production phase gets boring in the best possible way.

We help teams scope and ship AI integrations with the rigor they apply to the rest of their product. Let's talk.