I’ve posted more than once about the useless things in AI risk management. One of these is benchmark testing.
Benchmarks are useful for comparing models. But they have their limitations when you are trying to deploy something for your task that is part of your use case, in your environment, with your data, and under your constraints. Benchmarks are not a substitute for contextual testing.
I once asked a bank how they tested a coding agent. They replied that they used standard problems such as the traveling salesman problem, sorting etc. I said why did you do such useless testing? You could have saved your time by reading a paper benchmarking the model. Such testing told you nothing about how the agent would perform with your codebases, libraries and the organisation’s coding quirks.
Earlier in the year, I spent some time helping write a chapter in Singapore Computer Society’s AI Ethics and Governance Body of Knowledge on evaluating and assuring agentic AI systems. The chapter is called “Evaluation and Benchmarking for Agentic AI Systems: From Benchmarks to Assurance”. Co-authors are April Chin and Mia Hoffmann at Resaro.
It is now out. Here are some key points from the chapter:
A benchmark tells you what an agent can do. Deployment depends on what it will do, reliably and safely and at a cost you can live with, inside your own systems. Those are two totally different questions.
The reason is that an agent is not just a model. It is the harness around it - the planning loop, the tools that act on the world, the memory that carries mistakes forward, the connections into your systems. Classical evaluation was built for one model answering one prompt. It measures almost none of what an agent actually does across many steps.
Across many domains, most agents run only about ten steps before a human has to step in, and most teams still check them by hand. Evaluation is not keeping up with deployment. And you cannot scale what you cannot show to be reliable.
The chapter sets out the key layers to evaluate across, principles for doing it inside an enterprise, and a five-stage playbook for teams starting from nothing. The playbook does not start with public benchmarks. It starts with your own tasks.
Grateful to April and Mia for the collaboration, and to the Singapore Computer Society team and reviewers who made it sharper. Link in the comments.
#AgenticAI #AIRiskManagement #AIEvaluation #AIGovernance
Link (login required) - https://www.scs.org.sg/bok/ai-ethics-v2-1?document=b46188f7-523c-4332-9a5a-3ae006e0ac51
PDF print-out from SCS’ Book of Knowledge
