How to evaluate cost and quality in an AI workflow

A change that reduces model usage also needs to preserve the quality of the result. Evaluating cost and quality together means testing representative tasks, checking outputs, and accounting for the review or correction work people still need to do. That gives a more useful basis for comparing different approaches.

This guide covers how to build a test set, define what a good result means, measure the full cost of each option, and keep checking after launch. The examples follow a made-up property management company that uses AI to read tenant maintenance requests, assign a priority, and route each one to the right contractor.

Why cost and quality belong in the same test

Suppose the property manager switches the routing step to a smaller model. Model usage drops. If more requests now land with the wrong contractor, though, someone has to notice, reassign them, and apologise to the tenant. That work has a cost too, and it can outweigh the savings.

The useful measure is the cost of a completed, correct task. That includes model usage, the time people spend reviewing outputs, and the time spent fixing errors. Comparing options on that basis shows which one actually costs less to run.

Build a set of representative tasks

Start with real examples from the work itself. For the property manager, that means a sample of past maintenance requests, along with the priority and contractor each one should have received.

A good test set covers the range of what the system will see:

• Common cases. The routine requests that make up most of the volume, such as a dripping tap or a broken light.
• Hard cases. Requests that are vague, describe several problems at once, or need judgment about urgency.
• Edge cases. Messages with only a photo, messages in another language, very short messages, and forwarded email chains.
• High-stakes cases. Anything where an error has serious consequences, such as a gas smell, a water leak, or a broken lock.

Each example needs a reference answer prepared by the people who do the work. Where they disagree, settle it before testing. If the reference answers are inconsistent, the scores will be too.

Keep part of the set aside and use it only for final checks. If you tune the system against every example, it can look better on the test than it will on live work.

Handle the test set with the same care as the data it came from. Remove personal details you don't need, and store it under the same access rules as the source.

Agree on what counts as a good result

Write a short scoring guide before running any comparison. For each output, decide what gets checked:

• Is the category correct?
• Is the priority correct?
• Did it pick the right contractor?
• If it drafted a reply to the tenant, is the reply accurate and appropriate?
• Did it add anything that isn't in the request?

Then sort errors by severity. Marking a gas smell as low priority is a critical error. Sending a routine repair to a reasonable second-choice contractor is minor. A misspelled street name in a draft reply is cosmetic. Severity matters more than a single overall score, because two options with similar averages can differ a lot on critical errors.

Some checks can be automated, like whether a field matches the reference or whether the output is in the right format. Others need a person. A model can help grade outputs at larger volumes, but have people check a sample of its grading regularly. Two reviewers using the guide should reach the same score on the same output. If they don't, tighten the guide.

Measure the full cost of each option

For each option, record:

• Model usage per task. The amount processed and the number of calls made.
• Speed. How long it takes to produce a result, if timing matters to the workflow.
• Review time. How long a person spends checking the output.
• Correction time. How long it takes to fix an error, by severity.
• Escalations. How often the system hands a task to a person entirely.

Turn review and correction time into cost using a loaded hourly rate for the people doing it. Add model usage and infrastructure. Divide the total by the number of completed tasks, and you have a cost per task you can compare across options. Consistency matters more than precision here. Use the same method for every option.

Design choices worth comparing include:

• A larger model and a smaller one.
• A single call and several smaller steps.
• More reference material and less.
• With and without a verification step.
• Routing easy cases to a smaller model and hard cases to a larger one.

A worked comparison

The property manager compares two designs on the same set of past requests.

Option A sends every request to a larger model in a single call that assigns category, priority, and contractor.

Option B uses a smaller model first. It handles the requests it classifies as routine and sends anything urgent, unclear, or unusual to the larger model.

Option B uses less model usage, so it looks cheaper at first. The scoring guide decides whether it is.

The first check is critical errors. If Option B misses urgent requests more often than Option A, the comparison ends there. A slow response to a leak can cost far more than the usage you saved.

If critical errors are level, look at the rest. Say Option B makes a few more minor routing errors, and each takes a coordinator under a minute to fix. Add that review and correction time to Option B's cost. If it still comes out lower per completed task, Option B is the better choice. If the corrections eat most of the savings, Option A may be worth the extra usage because it's simpler to run and maintain.

Also look at how each option fails. A system that flags uncertainty and passes a request to a person is easier to work with than one that is confidently wrong. Read the failures one by one. They often point to a fix, such as a missing contractor category or an instruction that needs clarifying.

Keep testing after launch

Quality can shift after launch even if nothing in your own system changes. Inputs change, and providers update models. A few habits keep the comparison current:

• Re-run the test set whenever you change the model, the instructions, or the reference material.
• Re-run it when a provider updates or retires a model you rely on.
• Add new failure cases from live work to the test set.
• Review a sample of live outputs on a regular schedule.
• Track review time, correction time, and cost per task over time.

The same test set helps with other decisions too, from choosing an AI model for a business workflow to planning the ongoing cost of an AI system. Once it exists, it becomes the reference point for every change the team considers.