“We don’t want concepts!”
I’ve been helping train people for maybe a decade now. I remember that the first one I did was an MAS in-house training on model validation, maybe around 2016.
When I did my masters in knowledge engineering and another on financial engineering, it was something I probably thought as well. Especially for the hard classes like stochastic processes.
Interestingly, years of actually doing things (inspecting and building real models for different use cases, running investment risk for billions in reserves) made me appreciate the need to understand how things work. Developing AI models at a fundamental level over 3 years of hardcore PhD research, and writing the AI risk management guidelines for the financial sector has also made me appreciate the need for first principles for AI governance and risk management more.
And I’ve realised that it’s sometimes not clear what people mean when they say “concepts”.
Packed week again, and a tiring one. Four nights I was teaching AI risk to financial supervisors from across the Americas, starting at 9pm and ending at 1am because of the time zones. In between, I built an openly available online guide on the MAS AI Risk Management Guidelines (AIRG) for risk and compliance teams. And set out plans to do more such accessible stuff. And I took a long string of calls on gigs and training.
I’ve been hearing this point on “concepts” more recently. In fact, for all my classes, I start off by asking people whether they want the “why”, “what” or “how” and calibrate accordingly. I can do all three pretty easily at this point. Whether it is explaining the “why” or doing hands-on stuff with the “what” and “how”.
And I’ve realised some things. That learning is about a stool with three legs. On not overfitting when learning. And that this has something to do with my frustration with the “prompt and pray”, “prompt and awe”, and “prompt and parrot” malaise these days. More on these below.
The three legs
This point on not wanting to learn concepts is valid. But it conflates things.
It assumes that concepts are one thing. As if concepts were the thing you put up with before the useful part. And worse, some people have used it as shorthand for not wanting to understand what is going on. Or to throw a few use cases into a power point deck or a facilitated exercise and call it practical hands-on.
So last week I wrote about shallow and deep AI. This is the sequel.
On the three legs of what I call the learning stool.
When I ask a room whether they want the “why”, the “what” or the “how”, I’m offering them the three legs of the stool.
These aren’t three rungs on a ladder, with concepts at the bottom and the real work up top. They’re three legs. Take one away and the stool tips over.
In a class I draw the fundamentals live, by hand, and I write real code and run real demos in front of people. And I tell people that use cases are just simple collection of repeatable tasks and show how to assemble them.
Why, what and how, in the same session, because they aren’t in competition.
Practical and fundamental do not sit on opposite sides. Someone who only knows how to pitch the former and belittle the latter is not someone I can respect.
Technical is not fundamental
Being fundamental is the “why” and first principles. Understanding what’s actually going on underneath. What the task really is once you strip the use case name off it. What the data is, and whether it’s any good. How the system behaves, how it fails, and whether you’d notice when it does.
Being technical is the “what”. What the thing is made of and how it runs. The maths, the code, the architectures, the tools for evaluating what comes out. Hands on the parts.
You can be very technical and have almost no fundamentals. And vice versa.
You can know the architecture, wire up the pipeline, quote the paper, and still not say what the task really is, or what good would even mean for it, or how it would fail in the hands of the people who use it.
And the reverse. You can understand the task cold, know exactly what good looks like and how the thing will fail, and still not be able to wire up the pipeline or read the code that runs it.
It’s fine to know one better than the other. I respect that. Just don’t mistake one for the other.
Practical is not the opposite of fundamentals
Being practical is the “how”. Using it to get a real result in a real place. The workflow, the judgement, the deployment, the outcome for an actual team with actual constraints.
This is the one that gets mixed up the most. The word “practical” is usually doing two things at once.
One is tangible now. It produces something you can point at, it feels like doing rather than thinking, it gives you a result on the spot.
The other is useful later. You can do the thing again, in a situation you haven’t seen, and still get it right.
Just applying a tool to a use case scores full marks on the first and close to nothing on the second. Ditto for doing an exercise on a use case, if done without the right fundamentals or technicals.
So people learn to use a tool against a use case, get a clean output, and walk away thinking they’ve learned something they can use. They’ve learned one use case. The output looked good, but they never said what good meant, so they can’t even tell whether it worked because of what they did or in spite of it.
It’s overfitting, done to a person
There’s a name for this, and anyone who has trained a model before knows it.
Overfitting is when a model memorises the exact examples you trained it on instead of learning the pattern underneath. It scores beautifully on the data it’s already seen. Then you show it something new and it falls apart, because it never learned what generalises. We put a lot of effort into stopping models from doing this, because a model that’s overfit is close to useless the moment the world changes.
Just applying a tool against a use case is the same thing, done to a person. And that’s what I see as the “prompt and pray”, “prompt and awe”, and “prompt and parrot” malaise.
You fit the examples. You ace the cases you practised on. And you generalise to nothing.
And the layer people overfit to is the worst one they could pick. Tools change every few months. So what some people keep calling practical is often the least practical thing in the room. It works once, on the case in front of you, and leaves you nothing for tomorrow.
Where it actually does damage
How this stops being an annoyance and turns dangerous is when it comes to governance. And the cost doesn’t land on whoever skipped the concepts. Overfit while learning a tool and you waste an afternoon. Overfit while governing, and there are downstream consequences.
Think about what governance actually is. Knowing what the task really is, what the data is, how the system fails, what good would mean, who owns the risk, and whether anyone would notice when it breaks. Every one of those is a concept. None of them is a prompt or a use case.
Governance is almost nothing without the fundamental layer. And it’s worse than that, because governance is the part that has to generalise. Controls exist to hold up in the cases you haven’t seen yet. That’s the whole point of having them. Someone who has only ever fit the use cases in front of them governs nothing, because the next case is unseen and often unknown, and an overfitted understanding doesn’t carry over to it.
So someone who waves off concepts as impractical has, without noticing, waved off the whole of governance. There’s nothing left to govern once the concepts are gone.
And I’ve said it in my classes to both regulators and industry. If your capability building means sending people off for shallow hands-on with some use-cases, “prompt and pray”, “prompt and awe”, and “prompt and parrot”, then you have not only not built capability, you have made it worse.
I detest this.
#AI #AIFluency #AIGovernance #AILiteracy #Reflections #AIRiskManagement