How to choose an AI model for a business workflow

Selecting a model begins with the work it needs to perform. A useful evaluation uses representative inputs, agreed quality criteria, and the conditions the system will face in practice. Comparing those results alongside cost, speed, and deployment requirements helps a team choose an approach it can operate.

This guide covers how to describe the task, set requirements, test a short list of models, and record the decision. The example follows a made-up industrial parts manufacturer whose sales team receives requests for quotes by email, often with drawings and specification sheets attached.

Describe the job before looking at models

Start with a plain description of the task. For the manufacturer, it might read like this:

"Read each incoming quote request, including attachments. Extract the part numbers, quantities, materials, tolerances, and requested delivery date. Enter them into the quoting system for a sales engineer to review. Flag anything missing or unclear."

Then fill in the details that affect model choice:

• Inputs. Emails, PDFs, scanned drawings, spreadsheets. Which languages? How long is a typical request?
• Outputs. Structured fields, a category, a draft reply, a recommendation?
• Speed. Is someone waiting for the answer, or can it run overnight?
• Volume. How many requests a day, and how uneven is the flow?
• Consequences. What happens if a quantity or tolerance is wrong, and who catches it?

A one-page description like this keeps the evaluation focused on the work. It also gives everyone involved the same reference point.

Set your requirements and constraints

Some requirements narrow the field before any testing. Work through them early:

• Data handling. Can these documents leave your environment? Check customer contracts, privacy obligations, and internal policy. Some drawings may be covered by confidentiality terms.
• Location. Does the data need to stay in a particular region?
• Deployment. Will the model run through a hosted API, in a private cloud environment, or on infrastructure you manage?
• Capabilities. Does it need to read images, handle long documents, return structured output reliably, or call other tools?
• Speed and volume. Can it respond fast enough at peak volume?
• Cost. What usage cost per task fits the budget? Our guide to planning the ongoing cost of an AI system covers how to estimate it.
• Support. What happens when the model is updated or retired?

For the manufacturer, suppose some customer contracts require their drawings to stay within the company's own environment. That rules out sending those documents to a hosted API, at least for those customers, and puts open models on the short list.

Build the evaluation from real work

Public benchmarks give a rough sense of what a model can do in general. They can't show how it handles your drawings, your customers' email habits, or your part numbering. For that, you need a test built from your own work.

Gather a set of past quote requests with the correct extracted fields, prepared by the sales engineers. Include the hard ones: poor scans, handwritten notes on drawings, requests that mix several parts, and attachments in unusual formats. Write a short scoring guide that says which fields matter most and which errors are critical. A wrong quantity or material is serious. A slightly misformatted date is minor.

Test under the conditions the system will face. Use the same instructions, the same document processing, and realistic document quality. If the system will run at peak volume, test speed at that volume too. We cover test sets and scoring in more depth in how to evaluate cost and quality in an AI workflow.

Compare a short list in practice

Pick a few candidates that meet your requirements. For the manufacturer, that might be a large hosted model, a smaller hosted model, and an open model it could run in its own environment.

Run each one against the same test set with the same setup. Small adjustments to the instructions for each model are fair, as long as the comparison stays even. For each candidate, record:

• Accuracy on each field, and the number of critical errors.
• How often the output comes back in the right format.
• Time to produce a result.
• Usage cost per request.
• How it behaves when it's unsure. Does it flag the gap, or fill it with a guess?

Read the failures as well as the scores. A model that misreads tolerances on scanned drawings has a different problem from one that confuses two similar part numbers. The first might be solved by better image processing. The second might need reference data from the parts catalog.

Suppose the large hosted model scores best overall, and the open model comes close on typed documents but struggles with poor scans. One reasonable outcome is a split. The open model handles requests from customers with strict data terms, with extra human review on scanned drawings, and the hosted model handles the rest. Another is to improve scan processing and test again. The results tell you which conversations to have next.

Plan for running it

Choosing a model also means choosing what you'll maintain.

With a hosted model, the provider runs the infrastructure. You still need to plan for model updates and retirements, usage limits, and what happens during an outage. Keep a fallback in mind, even if it's routing requests to people for a while.

With an open model on your own infrastructure, you control the version and where the data goes. You also take on capacity planning, security updates, monitoring, and the people to do that work.

Either way, it helps to design the system so the model can be swapped. Keep instructions, test sets, and integrations separate from any one model. When a new model appears or a requirement changes, you can test it against the same set and switch if it does better.

Record the decision

Write down what you chose and why. A short decision record should include:

• The model and version selected, and how it's deployed.
• The evaluation results for each candidate.
• The requirements it meets, and any it meets only with extra controls.
• Known weaknesses and how they're handled, such as human review on certain document types.
• The expected cost per task.
• What would trigger a new evaluation: a model update or retirement, a change in cost, a drop in quality, or a change in data rules.

This record makes later decisions easier. When someone asks why the system uses a particular model, or whether a newer one is worth trying, the reasoning is written down and the test to settle it is ready to run.