In short
- In single-pass tasks, quality depends on the harness around the model, not on the choice of model: about 80% of the result according to WeAreFuture implementations.
- In long agentic tasks, the model makes the difference: over 20 steps, 95% success per step gives 36% for the whole task, 99% gives 82%.
- A harness does not carry over between models, which is why a cheaper model often ends up costing more.
- Before you choose a model, count the steps in the task, the cost of an error and the success rate per step.
In summer 2026 an argument flared up online that shows well the confusion around AI in companies. Some practitioners claim the gap between open source models and the best frontier models is now practically negligible. Others say there is a chasm between them. Both sides sound convincing, because both are right, they are just talking about different things.
This is not an academic dispute. It is a question your budget pays for. If the difference between models is negligible, you take the cheapest one and enjoy the savings. If it is huge, you either overpay or deploy something that does not work. And the real answer, for most business uses, is different from both extremes: the choice of model does not decide the result. The harness does, meaning the whole shell the model is plugged into.
That sentence has a second half, however, which most of the discussion skips, and it is the half that protects you from an expensive mistake: the harness is not portable between models. So “the harness matters more than the model” does not mean “you can swap in a cheaper model at no cost.” This article is about how to reconcile those two things and what follows from them for your decisions. I write from a practitioner’s perspective: WeAreFuture is a Polish AI consulting and implementation firm from Warsaw that builds agents and automations for mid-sized and large companies.
What is a harness in AI?
A harness is the whole shell that turns raw model weights into a working system. The term comes from the world of model testing (evaluation harness, like the open lm-evaluation-harness from EleutherAI). On its own, a model only generates text. The harness makes that generation turn into something that performs a task.
A harness consists of seven layers:
An AI harness is seven layers around the model: from instructions and context to evals and monitoring.
Instructions: the system prompt, domain rules, context files (like CLAUDE.md) that tell the model how to behave.
Context engineering: what goes into the context window: retrieval, compaction, long-term memory. This decides whether the model has anything to work with. Anthropic describes it as a separate discipline in Effective context engineering for AI agents.
Tools: tool schemas, the quality of their descriptions, the wording of error messages and the feedback loop that tells the model whether its action succeeded.
Agent loop: planning, retries, verification of the result, splitting into sub-agents, checkpoints.
Guardrails: enforced response structures (structured outputs), validation, a human in the loop where the stakes are high.
Scaffolding: sandbox, file access, code execution, i.e. the environment the agent works in.
Operations layer: routing between models, caching, observability, tests (evals) and regression monitoring.
The simplest metaphor: an engine from a Formula 1 car dropped into a regular production body, without the right gearbox, aerodynamics and telemetry, will not go any faster than an ordinary car. The model is the engine. The harness is the rest of the car and the whole pit crew. The engine alone does not win the race.
And that is the core of the argument we started with. When two companies give the same problem to two different models, but in two different harnesses, the difference in the result comes mostly from the shell, not from the model. That is why the same model can deliver a brilliant result for one team and a useless one for another.
You can see it in benchmarks. LangChain raised the score of its coding agent on Terminal Bench 2.0 from 52.8% to 66.5% by changing only the harness, with no change of model (LangChain, Improving Deep Agents with harness engineering). The authors of the paper “Stop Comparing LLM Agents Without Disclosing the Harness” (May 2026) show that on long-horizon tasks, changing the shell can shift the result more than changing the model, even to the point of reversing the ranking of models.
Why are both sides of the argument right?
Some see a negligible difference between models and others see a chasm, because of simple math: error propagation.
In a multi-step task, the success rate of the whole is the success rate of a single step raised to the power of the number of steps. The longer the task, the more small differences accumulate.
Over 20 steps, 95% success per step gives 36% success for the whole task, while 99% gives 82%.
Success rate per step | 10 steps | 20 steps | 50 steps |
|---|---|---|---|
90% | 34.9% | 12.2% | 0.5% |
95% | 59.9% | 35.8% | 7.7% |
99% | 90.4% | 81.8% | 60.5% |
99.9% | 99.0% | 98.0% | 95.1% |
This table holds the whole argument. A four percentage point gap on a single step (95% instead of 99%) turns, over 20 steps, into the difference between just under 36% and almost 82% success for the whole task. Hence the “huge difference”: on long, multi-step tasks the better model simply delivers and the weaker one falls apart. And hence also the illusion of a “negligible difference”: on tasks of one, two or three steps both models look practically identical, because the error has no room to build up.
The same math leads to a conclusion that matters more for business, and it is an argument FOR the harness. If success depends on two things, the quality of a single step and the number of steps, a good harness works on both. It reduces the number of critical steps. It inserts checkpoints and validation so that an error does not propagate further but is caught immediately. It resets the context instead of letting it rot. So the harness changes not only the base of the power (step quality) but also the exponent itself (the number and structure of steps). That is why a good harness on a weaker model can beat a weak harness on a better one.
Three classes of tasks: when does the harness decide, and when the model?
Instead of asking “who is right,” it is better to ask “when is who right.” AI tasks fall roughly into three classes, and in each the balance of power is different.
In single-pass tasks the harness decides the result, in long agentic tasks the model does.
Task class | What decides the result | Examples |
|---|---|---|
A. Single pass, narrow domain | Harness (about 80%) | classification, data extraction, RAG Q&A, summaries, simple text |
B. A few steps with tools | Roughly half and half | automations, ticket handling, quote generation, orchestration in n8n |
C. Long agentic horizons | Model | agentic coding, multi-hour research, autonomous loops |
The proportions in the table are a WeAreFuture estimate from implementations, not a research result.
Class A is single-pass tasks in a narrow domain. Here the harness dominates at around 80%, and a cheap or open model is entirely enough. The limit is not the model’s intelligence but the quality of the data and the retrieval. If the model gets the right context, anything from that shelf will cope.
Class B is multi-step workflows with tools. Here the balance of power is roughly half and half. What decides is the quality of tool use and the stability with which the model sticks to instructions across several steps. This is exactly the area we described in the difference between an agent and automation.
Class C is long agentic horizons: agentic coding, multi-hour research tasks, autonomous loops. Here the model dominates, because the error accumulation from the table above kills, and the better model wins outright. This is the field where supporters of the thesis “the model makes a huge difference” are fully right. It is also the field that moves fastest: according to METR (2025), the length of tasks that frontier models complete autonomously doubles roughly every 7 months.
And here is the punchline for the Polish market: the vast majority of projects in domestic enterprises are class A and B, not C. That is why the thesis “the harness decides the result” is true for business implementations, and the thesis “take the best model, because only that counts” is true for the research and development frontier. Both sides of the argument describe the truth, just from two different worlds.
The trap: why is a cheaper model not an automatic saving?
From the thesis “the harness matters more than the model,” many people draw a false conclusion: “if so, let’s swap in the cheapest model and pocket the savings.” That is a trap, because the harness is not portable between models.
Scaffolding tuned for a frontier model quietly assumes several things: a long context window, strong and reliable tool use, resistance to instruction drift over many steps, and sensible planning. Moved to a smaller open source model, it usually falls apart. To restore it you have to break prompts into smaller ones, add validations, shorten and simplify steps, write retry logic, and sometimes add fine-tuning. All of that is engineering work, and not trivial work.
As a result, at the typical volumes of the Polish mid-market, the engineering cost of bringing an open model to parity usually exceeds the savings on the API. A “cheaper model” can therefore turn out more expensive, only the cost moves from the API invoice to your team’s time. This does not mean open source is bad. It means the model decision is a decision about the whole system, not about a line in a price list.
The practical recommendation is simple: a frontier model on the decision layer, where single-step quality and long chains matter, and a cheap or open model on the mass work, where tasks are single-pass and high-volume. This is not an either-or choice, it is a matter of matching the model to the task class.
What does it look like in practice?
In our implementations we see it every time. In a product configurator built for a manufacturing company, the choice of a specific language model was one of the last and least important decisions. The value was created elsewhere: in the data model, in how state is stored and versioned, and in how the agent reaches for product data. If we swapped the model there for another from the same shelf, the user would not notice the difference, because the quality is decided by the shell, not the engine.
The same goes for an agent handling requests for quotation (RFQs), built on orchestration with integrations to company systems (ERP, CRM, knowledge base). Around 80% of the work there was integrations, loop design and validation, not the prompt itself or the choice of model. It is a textbook example of a class B task, where the harness decides the result. If someone had told that team to “deploy the best model and that’s it,” they would have started with the least important variable and skipped what actually creates value.
The common denominator is always the same: the result is decided by the engineering, data and process around the model. Exactly what we wrote about in context files and skills, because context engineering is one of the pillars of the harness, and in AI implementations from chaos to order.
Decision checklist for CIOs and CTOs
Before you decide “which model,” go through these questions. They matter more than the model name in a proposal.
How many steps does your task have? One, a few, or dozens? This immediately tells you whether you are in class A, B or C, and whether the model will be the bottleneck at all.
What is the cost of a single error? The stakes for an email summary are different from those for a financial decision. The higher they are, the more validation and the more important the step quality.
Do you measure success per step? If you do not measure, you do not know where you lose, and the conversation about the model is a conversation about impressions.
Do you have evals and regression monitoring? Without them, every change of model or prompt is a leap in the dark.
Is your scaffolding model-agnostic? Is changing the model a configuration change, or a rewrite of half the system? This question decides the actual cost of a “cheaper model.”
Where in the process can an error propagate, and do you have checkpoints? Places without validation are places where one bad step ruins the whole.
Do you have a context and memory layer? Without it, the model starts from zero every time, and you pay for it in quality and tokens.
What does your AI budget go to? If it all goes to model licenses and not to the shell, you are probably investing in the least differentiating part of the system.
If the answer to most of these questions is “I don’t know,” you have a ready list of things to sort out before you spend a zloty on a “better model.”
Don’t choose a model. Design a system
The argument of open source versus frontier is badly framed. The model is a variable, not a foundation. The foundation is the system around it: instructions, context, tools, the loop, guardrails and the operations layer. They decide whether the same technology produces a working implementation or an impressive demo that falls apart in production.
That is why in our work we start not with the question “which model,” but with the system design, and we keep a principle of independence from the model vendor. The model is matched to the task class at the end, when the rest is already designed. The same logic stands behind how we build the AI Engine, a shared company context used by people and agents: data, procedures and process first, then specific implementations, and the choice of model is one of the last, replaceable decisions. How it all fits into a broader AI transformation in the company we describe separately.
If you are facing a decision on where to invest your AI budget, this is a good moment to look at the whole system, not just the model name. See our AI consulting and strategy or how we run AI implementations, and book a call, and we will show you where the value lies in your case.
Frequently asked questions (FAQ)
What is a harness in AI?
A harness is the whole shell that turns a raw model into a working system: instructions and context files, context engineering (retrieval, memory), tools, the agent loop, guardrails, scaffolding and the operations layer (routing, evals, monitoring). The model generates text, and the harness makes that generation turn into something that performs a task. For most business uses it is the harness, not the model itself, that decides the result.
Does the choice of model matter at all?
It does, but it depends on the task. On short, single-pass tasks the differences between good models are small. On long, multi-step agentic tasks the better model wins outright, because error accumulates exponentially. The model is also the base the harness stands on. So the right answer is not “the model doesn’t matter,” but “for most business implementations the harness decides, and the model is matched to the task class.”
Is an open source model enough instead of a frontier model?
For many single-pass tasks (classification, extraction, RAG) often yes, because the limit there is data quality, not the model’s intelligence. For long agentic tasks usually not. Watch out for the trap: a harness tuned for a frontier model does not transfer directly to a smaller model, and the cost of bringing it to parity can exceed the savings on the API.
Why does the same model give different results in different tools?
Because the harness differs. The same model plugged into a better shell (better context, tools, loop, validation) gives a radically better result than in a weak one. LangChain raised its agent’s score on Terminal Bench 2.0 from 52.8% to 66.5% by changing the harness alone, with no change of model.
What is context engineering?
It is taking care of what goes into the model’s context window: the right retrieval, compaction, long-term memory and order in what the model has at its disposal. It is one of the pillars of the harness and a topic we develop separately in the piece on context files and skills. Without good context, even the best model works on barren ground.
Will switching to a cheaper model reduce costs?
Not always. The API invoice itself may drop, but if the cheaper model requires rewriting the scaffolding, adding validation and shortening steps, the cost moves to the team’s time and can exceed the savings. At typical mid-market volumes, do the math before you switch. It is a decision about the whole system, not about a line in a price list.
What does the success of an AI agent depend on?
On two things at once: the quality of a single step and the number of steps. The success rate of the whole is the success rate of a step raised to the power of the number of steps, so small differences accumulate exponentially: over 20 steps, 95% per step gives 35.8% for the whole, and 99% gives 81.8%. A good harness improves both factors: it raises step quality, and it shortens the chain of steps and protects it with checkpoints so that an error does not propagate.
How do I choose a model for an implementation?
Start with the task class and the number of steps, not with general benchmarks. For high-volume, single-pass tasks a cheap or open model is enough. For the decision layer and long chains, reach for a frontier model. Design the system first (data, context, tools, loop), and treat the choice of model as one of the last, replaceable decisions.
Sources
LangChain, Improving Deep Agents with harness engineering - agent score on Terminal Bench 2.0 from 52.8% to 66.5% with no change of model (gpt-5.2-codex).
Zhang et al. (2026), Stop Comparing LLM Agents Without Disclosing the Harness, arXiv:2605.23950 - the harness as a stronger determinant of an agent’s result than the model on long-horizon tasks.
METR (2025), Measuring AI Ability to Complete Long Tasks - the length of tasks completed autonomously by models doubles every approx. 7 months.
Anthropic, Effective context engineering for AI agents - context engineering as a discipline of building agents.
EleutherAI, lm-evaluation-harness - the origin of the term “harness” in model evaluation.
Error propagation table: WeAreFuture’s own calculations (success rate per step raised to the power of the number of steps). Proportions of classes A/B/C and the share of work on the quoting agent: data from WeAreFuture implementations (2026).
