The Part That Doesn't Change
Harnesses and AI governance
How to cope with change in AI governance.
I spent an hour with a room of senior folks from a major insurer earlier. My presentation was titled “Some thoughts on scaling AI governance”. Near the end, someone posed a question that’s on the minds of most folks. But with a new analogy.
In most of finance, he said, things are quite stable. First line takes the risk. Second line checks it. Third line audits it. Like positions on a football field. With AI, the field itself is shaking. The technology shifts. The regulatory standards shift. The models change every few months.
So how do you govern something that will not hold still?
Let me jot down how I answered it. And I wanted to connect my answer to the room to something that has been happening in research. Which I did not have time to go into there.
Be boring, don’t chase the AI model
My first instinct, as usual, was a boring one.
Whether the field is shaking is a matter of perspective. And many things still hold.
You do not anchor your governance to the thing that keeps changing. For AI, it’s the AI model, the LLM, the third party AI provider that does not tell you they made a change. You anchor it to the thing that is not moving.
And the task and the risk management system around the task does not move. Fraud detection is fraud detection. Insurance underwriting is insurance underwriting. The task defines what a good answer looks like. Not the AI model. And what a good answer looks like defines how you decide. None of that changes when the AI underneath changes.
Against the task, the model can be the swappable part. From the hopeless Llama 3 to the mythical Mythos - if you have a good system in place - e.g., a good evaluation and test set to measure what’s good, good processes and a solid platform to evaluate and test against, your endgame is the same. It does not matter that AI keeps changing.
The task remains the same. The system surrounding the task can also be the same.
In the room, I said this could be viewed as the equivalent of a harness for AI governance and risk management.
And folks have moved on from prompt to harness to loop engineering LLMs. The only one I think that has some substance is harness engineering. The rest are just hype. (Sidenote: The room also asked how to keep up in terms of skills sets. I said do not hire someone to train your stuff in prompt engineering because it’s a waste of time. Hire someone to train your staff about how to think with AI.)
And for the harness, we have had one in risk management far longer, we just never named it. It is the set of policies and processes, procedures, protocols, evaluations, testing sets, platforms, boundaries and controls we wrap around whatever we need to manage the risk of. Need not be a model. Could be a derivative. A new product. But the harness, if you have seen it across all these developments, can be remarkably stable.
And it compounds. Example, the more reps you get running evaluation and testing, and expanding your testing datasets, the faster you absorb each new model that lands - because you already know what you are testing it for.
Capability and risk are just different sides of the same coin
Separately, a bunch of papers this past quarter has been circling the exact same idea - from the other direction. Not how to govern a moving target, but how to squeeze more out of one. And in every one of them, the model is held completely still.
Meta-Harness searches for better scaffolding automatically, weights frozen. On a classification task it jumped in performance while using four times fewer tokens, better and cheaper (and probably faster too), without thinking about model. (arxiv.org/abs/2603.28052)
The Harness Effect did the same, but for cost. Across six models and twenty-two tasks, changing only the orchestration cut cost per task by about 30-60 percent at equal quality - and moved the bill more than the entire gap between the cheapest and most expensive model. (It is their own harness, and they are probably selling something here, so read with a pinch of salt.) (arxiv.org/abs/2607.06906)
From Model Scaling to System Scaling says the same. Once a model is good enough, the next gains come not from the mind inside the machine but from the system around it - how it remembers, what it retrieves, how it routes work, how it checks itself. The frontier, it argues, is that surrounding discipline, not the model. (arxiv.org/abs/2605.26112)
There are others. And there will be more.
Read together, they say one thing. Capability is becoming a property of the system, not just the model. The model will keep changing. But the harness is perhaps where the leverage now lives. Especially for everyone else but the frontier companies.
My point above on governance and risk management has no papers. But I don’t think it’s very different. Same coin, just different sides.
Nothing new under the sun
Finance has done this before. It always does.
Every so often a new instrument arrives that no one has priced - portfolio insurance, credit derivatives, structured products, the first trading algorithms. Each time, the instrument is new and the discipline around it is not. You size the exposure. You set the limits. You monitor. You keep the authority to stop. The thing in the middle keeps changing. What we wrap around it barely does. AI is just the newest thing in the middle.
Take structured derivatives. A bank invents a product with a structure no one has priced. There is almost never enough data to validate it, the data sources do not exist yet.
So what did they do? They did not refuse to sell it. They also did not pretend to a certainty they did not have. They deployed it inside boundaries. How much you can sell. To whom. Which thresholds you watch. When you pull the rug.
AI is the same shape of problem. You can train and test on the data you have. But until it hits real deployment, you do not truly know how it will behave. So you deploy it the way finance has always deployed the genuinely new, in a proper harness, with conditions, monitored, with a way to stop it.
The harness is what holds those conditions. The model is just what is sitting inside it this month.
First principles, not the method of the month
The trap is to govern the model by its techniques. Because the techniques churn fastest of all.
I get asked constantly: there is no settled explainability method for generative AI, none for agents, so how do I use one for underwriting or fraud?
Here is the reframe. Explainability is not about explainability. It is about understanding. You do not need to track whether the method of the moment is SHAP or some mechanistic-interpretability paper from last week. You need to know what you want the understanding for - who has to understand what, to make which decision. Get that right, and a good evaluation set will often teach you more about the system than any named method would.
Same for everything else. You do not memorise every metric. You ask what it measures, and whether that is the thing the task actually cares about.
This is also what the harness is made of. The methods - the explainability technique, the eval metric, the guardrail of the week - are just parts you slot in and swap out. The harness is the frame that decides what each part is for.
Ask from first principles and the churn stops mattering, because the questions underneath are stable even when the methods on top are not.
And no - please do not go buy another prompt-engineering course. That is not the skill. The skill is learnable, and it is not that hard. How do models actually work. Where are their edges. What can they not do. If someone says they want a language model to forecast markets, you should be able to say: it is trained on text, show me the evals, show me what it beats. Don’t bullshit me. That question does not need a technical deep dive. It needs a mental frame.
But the danger moved too
I would not read those papers and only feel reassured, though. If the harness is where the capability now lives, the harness is also where the failures now hide.
A stale note in the model’s memory is more dangerous than no note - it hands the agent confidence at the exact moment it should stop and check. A helper sub-agent can fail quietly, because its answer sounds plausible and nothing downstream checks whether it is true. Put a model in charge of watching another model, and you have grown a second thing that can be wrong.
Scale the harness without scaling your ability to see into it, to stop it, and to answer for it - and all you have built is a greater risk surface.
So, the shaking field
The model will keep changing. So stop nailing your governance to it.
Build the harness - the durable set of controls, checks, tests and boundaries around whatever model is inside it this month - and the shifting technology becomes something you swap through, not something that swaps you out. The engineers scale the harness to get more out of AI. We scale it to stay in control of it.
I do not have this fully worked out - if I did I would be rich, and I would be selling it. But this is how I am starting to think about it.