The leaderboard is the wrong place to start
A model that tops a benchmark can still be the wrong choice. If it cannot run where your data lives, cannot use the data you are allowed to give it, or cannot be looked after by your team once we leave, its score does not matter.
So we settle the boundary first and choose the model second.
Five things to settle first
Where it runs
Your Macs, your server room, your cloud account, an air-gapped room, or a remote service you have approved. Each rules models in and out.
What it may touch
Which material the model may learn from or read, and what may leave the environment: outputs, logs, prompts, temporary files. Decide what is kept, returned or deleted at the end.
Who can use and change it
Who runs it day to day, who can change it, who reviews its work, and who takes over after the first delivery.
What the machine allows
Memory, speed, how much text it must read at once, storage, and how many people will use it at the same time.
What must be left behind
Test examples, run records, notes on known limits, a rollback path and operator notes, so the team that runs the model understands it.
How the boundary changes the choice
These answers often decide more than any benchmark. They settle:
- whether the model runs on your own machines or in an approved remote service
- whether it must be compressed to fit the hardware
- whether an adapter on a smaller model is better than replacing the model outright
- whether testing should focus on speed, accuracy, format or how easy it is to review
- whether a rollback path must be proven before anyone uses it
When we trained a model to operate Mac software, we chose Qwen3.5-9B for exactly this reason: it can read a screen and still run on a single Mac.
The order we work in
- The boundary. Where it runs, what it may touch, who looks after it and what the machine allows.
- The shortlist. Only the models, adapters and runtimes that fit inside that boundary.
- The tests. Your task examples, the known hard cases, speed and review criteria, run against the shortlist.
Choose from the models that can live inside the boundary. Drawing the boundary after the model has been chosen means starting again.