Benchmarks are useful for comparing models. But they have their limitations when you are trying to deploy something for your task that is part of your use case, in your environment, with your data, and under your constraints. Benchmarks are not a substitute for contextual testing.

I once asked a bank how they tested a coding agent. They replied that they used standard problems such as the traveling salesman problem, sorting etc. I said why did you do such useless testing? You could have saved your time by reading a paper benchmarking the model. Such testing told you nothing about how the agent would perform with your codebases, libraries and the organisation’s coding quirks.

Earlier in the year, I spent some time helping write a chapter in Singapore Computer Society’s AI Ethics and Governance Body of Knowledge on evaluating and assuring agentic AI systems. The chapter is called “Evaluation and Benchmarking for Agentic AI Systems: From Benchmarks to Assurance”. Co-authors are April Chin and Mia Hoffmann at Resaro.

It is now out. Here are some key points from the chapter:

A benchmark tells you what an agent can do. Deployment depends on what it will do, reliably and safely and at a cost you can live with, inside your own systems. Those are two totally different questions.

The reason is that an agent is not just a model. It is the harness around it - the planning loop, the tools that act on the world, the memory that carries mistakes forward, the connections into your systems. Classical evaluation was built for one model answering one prompt. It measures almost none of what an agent actually does across many steps.

Across many domains, most agents run only about ten steps before a human has to step in, and most teams still check them by hand. Evaluation is not keeping up with deployment. And you cannot scale what you cannot show to be reliable.

The chapter sets out the key layers to evaluate across, principles for doing it inside an enterprise, and a five-stage playbook for teams starting from nothing. The playbook does not start with public benchmarks. It starts with your own tasks.

Grateful to April and Mia for the collaboration, and to the Singapore Computer Society team and reviewers who made it sharper. Link in the comments.

#AgenticAI #AIRiskManagement #AIEvaluation #AIGovernance

Link (login required) - https://www.scs.org.sg/bok/ai-ethics-v2-1?document=b46188f7-523c-4332-9a5a-3ae006e0ac51

PDF print-out from SCS’ Book of Knowledge

Agentic Evals Clean

402KB ∙ PDF file

Download

Download

What does it mean to be 'good' at AI?

Good at prompt engineering? Able to use an AI tool well? An expert at deploying AI systems? Able to build and train a foundation AI model from scratch?

Perhaps none of these things. Not for most people, in most jobs.

Being good at AI could be both simpler, and harder.

Stephen Tracy and I built Job AQ as an experiment to think about this. A free and short behavioural assessment to understand how well you actually work with AI, not what you know about it. In contexts that are relevant. The scenarios are drawn from archetypes that match your job.

Here's what you get from it.

Before: A feeling. "I'm pretty good with AI."
After: A score, and blind spots likely to trip you up.

Before: No real idea where you stand.
After: Your 4 behaviours side by side, where you are strong and where you are weak.

Before: Guessing where AI would even help in your work.
After: A skills map with the opportunities.

Before, for a team: Adoption numbers that tell you nothing.
After: Who can be trusted with AI, and where the blind spots are.

Take it once and you have a baseline. Take it again in a quarter and you have a before-and-after.

Free. Fast. Try it at jobaq.me.

The free ebook below explains "The Whole Elephant" (based on my old illustration). In the carousel below.

#AI #AIFluency #FutureOfWork #AIRiskManagement

Jobaq Ebook Extended

825KB ∙ PDF file

Download

Download

Someone saw one of my LinkedIn updates last week and asked if I had gone back to a full time job. No way, I said. I’m pretty much still a Grab or Uber driver by another name.

A heavy week. A discussion with the risk folks of a major insurer. Two runs of my AI risk management course for a banking association. A full day of a similar thing for insurance professionals. A few new engagements agreed before the week was out.

Interestingly, I’ve started to explain what I do now, not what I used to do. A lot of art. A lot of training. Lots of building. A fair amount of writing. From AI governance to financial regulation to the deep technical stuff.

I’m essentially a gig worker. Someone needs training. Or a piece written. Or some other odd job. And I’m there. If the destination fits.

Some reflections on independent life. And what gig work has to do with portfolio theory.

From Grab to portfolio theory

Much of what I do is not so different from driving Grab or Uber. But I know the risks of this kind of life. And so I’ve reached back to an earlier part of my career to manage these risks. I used to run investment risk for Singapore’s foreign reserves. It turns out the same rules apply to a very different portfolio - my own independent life. So 5 simple reflections from this perspective.

Never let one client own you.

Concentration risk is the enemy of an investment portfolio. It’s also the enemy of independence. The moment you have a single paymaster, everything bends to that relationship. You’re a pseudo-employee without the benefits of a real job.

So I spread it broad on purpose from the start. This week alone - a major insurer, a banking association, an insurance body, and a handful of new engagements agreed: a governance institute, a university, an international regulator programme. My LinkedIn now has close to 10 live affiliations. And a few more coming soon, including an adjunct gig teaching AI financial forecasting.

And I tell people I don’t just do training. I can help write and build too. And I don’t just do AI governance. I can do financial and investment risk. And a variety of financial regulatory topics. And I can go as deep into AI technicals as you want me to.

Charge the risk premium.

Higher risk, higher yield. Grab has peak pricing. Every gig has its own return and its own risk, and the price should track both.

I try not to judge a gig by the fee in front of me. I think better to price it across its full lifetime. Some work pays nothing or little now and pays much more later. Some work pays well now and costs you your freedom. And occasionally, there are the requests that just want to borrow what you have for free. Forever. For something called exposure that I do not care for.

I’ve become a lot clearer-eyed. And I try not to confuse being busy with being paid.

Have some dividend-yielding assets.

Much of what I do only pays when I am in the room, and it stops the moment I stop. That’s the biggest issue with such gig work. You can be fully booked and still be selling hours one at a time, forever.

So I have been building the other kind. This past week my ebook store went up. At learn.simplyboring.ai. 5 ebooks now. Another 10 in the works. Write it once, sell it many times. It is slow and unglamorous. But an ebook is the closest thing I have to an asset that pays dividends while I am doing art.

Hold something uncorrelated.

Every portfolio needs a position that does not move with the rest. Mine is the art. It pays almost nothing, but that’s not the point. This week, tired after four days of delivery, I started a whole new watercolour style anyway - because I cannot keep still, and because painting is the one thing I do that has nothing to do with AI.

I have no illusions that the interest in AI will last forever. Everything cools. The frameworks get commoditised, the novelty wears off, someone builds a tool that does the boring parts. The AI side of me will become irrelevant one day. When that day comes, I hope the art, the platforms, and the odd non-AI thing can help me hedge.

Rebalance when the work comes.

My life is now full of peaks and troughs. Sometimes, there are too many gigs. Sometimes, close to zero. And so when the email comes, the instinct is to grab all of it. Partly greed, partly the old fear that it might dry up tomorrow.

Now that I am at month 8, I can see a clear need to rebalance. I cannot keep taking every gig at the same price. I need to do fewer, and charge more for the hard ones. And say ‘no’ more often. Which I have now realised takes some skill. Even Grab drivers decline gigs, or wait for peak pricing. And the irony is that I have not rebalanced my own investment portfolio for months. And that’s also something one needs to keep watch on when independent.

So, a gig worker

A Grab driver and an independent like me are not so different. Both are just managing a portfolio. Mine is now 2.5 companies - Quaintitative (I know, confusing name) for the work, Simply Boring AI for the products, and a stealth one I am building with Stephen Tracy.

Almost 8 months in. The portfolio is taking shape. I still cannot keep still. And I don’t think that is going to change.

#IndependentLife #Quaintitative #PortfolioThinking #AIRiskManagement #Transitions #Reflections

At a board briefing, a director from a financial institution asked me a simple question - what do I actually need to know? I don’t think I answered that question well.

First, I did not have a clean answer to point him to. There has been plenty written for boards in general. I could answer his questions with what was in these publications, but that would not have been totally correct for finance. Second, answering it in detail takes a bit more than can be contained in a simple answer.

The Singapore Institute of Directors also put out its AI Guide for Boards in Singapore recently - a broad and generous map of the whole AI question a board faces. Strategy, value, talent, resilience, governance. Not a bad place to start.

What it said is generally applicable. But the financial sector’s risk management discipline, built over decades and paid for in real losses, has its own demands.

When the SID guide does reach into the financial sector, it reaches for the FEAT principles - Fairness, Ethics, Accountability, Transparency. Good principles. But principles are all they are. They tell you what a good outcome looks like. Not how to get there.

I had a hand in Singapore’s first set of AI risk management guidelines for the financial sector - the MAS AI Risk Management Guidelines (AIRG).

As the SID guide did not mention the AIRG, I decided to try to use this opportunity to answer the director’s question the way I know best - written, and through a financial-sector lens. This book uses the SID guide as a starting point (no point re-inventing the wheel), and connects it to the AIRG.

It is deliberately boring.

It starts with the five core areas in the AIRG that a board has to be responsible for, and the 5 questions I had shared with the board.

And then I go deeper.

It is just a start. So no matter how this got into your hands, feel free to pass it on.

The last page tells you how to reach me if you would like to argue with me. Tell me where I am wrong. I would like to learn more on this.

Just subscribe for free to my Substack to get access to the full ebook. Details in the email when you subscribe.

#AIRiskManagement #AIGovernance #AIBoards

Excerpt below.

Excerpt Ai Risk Management For Directors

1.47MB ∙ PDF file

Download

Download

I’ve been “unfocused” for many years.

Over the years, I’ve been interested in writing, speaking, building, teaching, researching and painting. Aside from finance, I have spent significant time in the creative industries and academia. And a number of side hustles - selling art prints, doing commissions, even selling Nokia mobile covers on EBay.

In a full time job, that’s usually a minus. You either get viewed as unfocused, or get asked to do things outside your job scope.

But now as an independent, it’s great.

A week of delivering and building. Several talks across different rooms and audiences. From one for the Singapore Venture Capital Association to another for foreign public servants for an SMU executive education programme. In the evenings, I am writing a book on boring questions that directors should ask on AI risk management. And building an AI governance game for an upcoming course, to address feedback from my learners. And at the end of the week, a conversation with an old friend and artist sparked a new production idea. Also managed to do more art this week.

The diverse range of things I did in the past made me grateful that I do not have to rely on anyone else to write, speak, build, teach, research or paint for me now. And led to some reflections on how something that was a bane when I was in a full time job is now a boon.

Focus and a full-time job

Full time jobs don’t usually like people who do many things. They need to box you. The performance review needs a focus area. There is always a risk of someone thinking that you are slacking off by working on your own interests.

Until someone needs a photographer for the work event. Or an illustration for the retreat. Or a creative take on something nobody else can handle. And suddenly the breadth that was supposed to hurt real work becomes useful. Because you can do it and nobody else can.

Over the years, I learned to keep work, and my many other interests separate for this reason. There’s no point. The institution will use your breadth whenever it’s convenient and brand you unfocused whenever it isn’t. Both at the same time. Without seeing the contradiction. There’s no malice. It’s just the way the world works.

Another useful lesson from those years is to draw the line based on relationships. You don’t say yes to everyone. Neither do you say no to everyone. You help those you like and trust. For the rest, no is the default.

The lack of focus is now liberating

As an independent, the stack helps.

This week I stood in front of a room of venture capitalists and talked about a world I do not know deeply because I have touched it once or twice during my frequent wanderings. I sat with an old artist friend and helped him think through using AI for a creative project, because I happened to do both art and AI. And instead of making yet another deck for a course, I built a small game, because I can build, design and understand the governance content at the same time. And AI has made all of these easier.

A few things have become clearer, now that the breadth is finally mine to use.

The “unfocused” label was never really about me. It was about a box I did not fit, and someone else’s discomfort with that. I half-believed it for years. I don’t anymore.

And I am not sure “unfocused” was ever the right word. Spread out, sure. But I gave each of them real time, not spare weekends. I am not brilliant at any of them - the painting is still lousy half the time - but none of it was dabbling.

The “unfocused” years turned out to be the ones that saved me. The side hustles, the detours, the odd skills nobody asked for - they may have cost me something in a career, but they are exactly what I get to use for myself now.

One thing has not changed. I still draw the line by relationships - I help the people I like and trust. What changed is another bonus of independent life. The default used to be one option - no. Now I can also use money as friction.

#IndependentLife #RojakLife #AIRiskManagement #Art #Transitions #Reflections

Nit-picking has been an occupational hazard for me since forever.

I recall someone from my team coming to me once. The number of tracked edits I had inflicted on her paper was causing her some distress. My answer was not to worry. I was doing the same to everyone else in the team. I assured her that her appraisal would be fine.

I’ve a slight streak of OCD in me. And I suspect it’s time for some karma. Worrying about small things when working a full-time job, especially in MAS, is generally a positive thing. Especially when you are drafting acts, regulations, or clearing contracts involving significant stakes.

And this week was a crazy bad one for this OCD. A talk for the leadership of a Ministry. A briefing to the board of an insurer. A talk for civil service directors doing a leadership programme. Three training sessions. One for a digital asset company. Two for senior leaders of a bank coming to Singapore for an executive leadership programme. Contracts after contracts. The launch of the JobAQ platform I did with Stephen Tracy. I was so tired at the end of a week I crashed and napped and missed a night call with someone from Toronto discussing training for some international regulators.

As an independent, I find the habit of worrying too much to be a bad thing. The stakes are way smaller. But I still worry before every talk. I read every contract twice, pass it through AI, then read again. And then worry about what the other party will think about my pushback. I check the demo the night before and again the morning of. I worry about the increasing number of ‘nos’ I say. These days I even have a niggling worry that something I wrote will be seen as AI generated.

But I think I am starting to learn how to deal with it. Sharing how it looks from the inside of independent life below. And my five rules for dealing with it.

The worry before and after every room

The engagements that stressed me out for the week.

Before the session for the leadership of a Ministry, I kept wondering why my deck, which came from my boring perspectives in the financial sector on AI governance would be relevant. I was quite happy that I connected them to Dr. David R. Hardoon who had a broader and more exciting perspective than me and that he would go first. Though I had secretly wished that they would feel there was no need for me.

Before the briefing to board of the insurer, I was worried about the relevance of my slides to their world. I’m not and have never been a director of a board. While I had staffed the MAS board for two years, that was mostly administrative. The session went smoothly. And there was one win. Someone said that a pre-read I wrote and shared was certainly not boring. But my internal spidey sense told me I had not given them exactly what they needed.

The two sessions for the senior leaders of a bank coming to Singapore for an executive leadership programme was worrying in a different way. The material was familiar to me. But it was a crowd that needed the material to be in Mandarin. There were translators, but conducting training through an intermediary is usually challenging. I tried using my half past six Mandarin (my entertainment diet is usually Mandarin shows on TVB, iQiyi and Viu after all). Was that a win or lose? I was not sure.

Even someone asking for my bank account for the honorarium of a paper on evaluation and testing for Agentic AI I wrote worried me a little. Because that meant the paper was done, and I worried if I had checked it enough.

By the end of the week I was in a heck-it mood. And I told the room of civil service directors doing a leadership programme that my aim for the 30 min talk was to make AI boring to them. And that my secret wish was that AI would become boring and fade into the background so I would not need to do any more of these panels and talks.

And I filtered contracts that could have led to some risks. One only paid for delivery but buried deep in its agreement was one line that would have transferred the rights to my IP to them. And there were other more minor things that needed to be caught.

This is what independent life feels like from the inside. Most probably think it’s freedom. From emails from bosses. From KPIs. From worry.

That’s not the case. All the worries remain. But all of it is yours alone to bear. No team to catch the error. No colleague to sanity check the piece. No institution to absorb the mistake.

So you check. Then you check again. And check one more time.

What I think one can do about it

First. See the stakes for what they are. Most of the time they are not as high as the worry makes them feel. The habit of treating everything as high stakes is a residue of environments where things actually were. Most things aren’t. Be clear about the real stakes. Don’t imagine them.

Second. Diversify. As much as possible. When one thing carries all the weight, the worry amplifies. Spread the load across enough engagements, projects and relationships and no single one becomes existential. It’s not just money that allows one to walk away. Choices help too. This is also just risk management 101.

Third. Detach from the outcome. Deliver the best you can. Then let go. The worry almost never predicted the outcome accurately anyway. The rooms that stressed me most often went fine. The ones I felt relaxed about sometimes went flat. That latent worry is a bad forecaster. Stop consulting it for predictions.

Fourth. Know your redlines. Not everything in a contract or a conversation needs to be negotiated. But some things do. Being clear early about what you will and won’t accept turns anxious reviewing into decisive filtering. For engagements themselves, use money and relationships as the filter. Calibrate your redlines of what you will do and not do based on how much you like the person vis-a-vis the remuneration. The more you like the person, the less money matters. And vice-versa. The worry shrinks when you know exactly what you’re protecting and why.

Fifth. Lose yourself in something that has nothing to do with any of this. For me it’s a watercolor every night before bed. One hour where presence is required and the worry has to wait. You cannot paint and worry at the same time.

#IndependentLife #Worries #AIRiskManagement #Transitions #Reflections

Try it.

#IndependentLife #Worries #AIRiskManagement #Transitions #Reflections

A director asked me a question at my briefing to a board yesterday. So what do we need to know about AI to do our jobs? I don’t think I answered it to his satisfaction. But what I’m sharing below might help you to answer it for yourself.

It starts with an elephant illustration I drew more than a decade ago. I dug it out and made it the motif for an app I built with recently.

Why?

Remember the six blind men who meet an elephant? One grabs the trunk: “a snake.” One touches the ear: “a fan.” One hugs a leg: “a tree.” All equally certain. All equally wrong.

I suspect that’s how AI feels to many of us right now. We each touched one part of this broad and deep thing. A few prompt tricks. One good demo. One project that went wrong. Some AI development in the news that seems like science fiction. But do we know the whole animal?

Certificates prove you sat through a course. Using an AI tool is, well, just using a tool. Quizzes prove recall. A prompt engineering course teaches you zilch about AI. Just how to type things into a text box. The snake. The fan. The tree. Not the elephant.

Job AQ is an experiment by Stephen and me. A simple (ok, maybe not that simple) behavioural assessment of how you work with AI. Not what you know about it. Scenarios drawn from your own job.

You don’t just get a score out of 100. You see where you stand on 4 dimensions - understand AI, work with AI, evaluate AI, own AI risk. Where your opportunities are, given your job archetype. And the gap between what you believe about yourself and what you actually did. So far, that gap surprises almost everyone.

The video below walks through the whole thing. (Yes, I know I look a little shabby, but it’s been a really long week.)

It’s at jobaq.me. Try it. Then send it to the five other people holding your team’s elephant.

#AICompetency #AIFluency #AI #JobAQ

~300 watercolors in one video. And a question on authenticity.

The past week I laid out every watercolor I have painted in the last two years. Just over three hundred of them. I did it to make a short video to see how my style has changed and try to get better. But somewhere in the middle of compiling it, I started thinking about authenticity and AI.

It was a full week otherwise. I finished my July series of AI risk management courses for the Association of Banks in Singapore - three runs, and by the end the feedback had flipped from “too technical” to “we want more”. I enjoyed an evening talk on time series forecasting at SMU for a mix of Singaporean and Korean masters students. A paper on AI and the workforce co-authored with ATA that went out, that I also enjoyed writing. A pile of slides for the weeks ahead. A number of training proposals - requests that interestingly came through my website. And, as always, a watercolor most nights before bed.

But it was the menial process of scanning three hundred paintings (which I did not enjoy at all) that sparked some thoughts.

Proving authenticity with AI is kind of stupid

You cannot fake a real physical watercolor piece. The water goes where it wants. The paint shifts in a way no one fully controls. It is impossible to do the same exact piece of the same scene with watercolors, even if you used a robot. In a world where almost nothing is provably human anymore, that turns out to be fairly uncommon.

And it made me think about writing, because judging authenticity in writing is hard. And somehow or other, it has become a lightning rod.

You cannot prove you wrote something by hand. Writing that we consume these days is rarely physical. Even if you sat down with pen and paper - you would still type it in to post it. Nobody handwrites a page, photographs it, and uploads the photo. (For now at least. Maybe someone will have some ideas after this.)

Whatever physical act happened, if it happened at all, becomes irrelevant the moment it goes online. No one can tell you who or what put the words there.

Substack recently started scanning posts with an AI detector called Pangram, and it did not go down well. The idea is to hand back some of that lost proof - to score how likely a piece was written by a machine. With a machine (!?).

I understand the instinct. But it is solving the wrong problem, doing it badly, and with a machine that could be generating slop itself.

Start with badly. Writers began getting flagged on their own work that’s theirs. Not AI.

I call this using AI slop to find AI slop.

We think we are solving the problem of AI slop. But Pangram itself may be slop. Biased slop that may fall hard on people who write in a certain way, such as non-native speakers of English.

And even a perfect detector would be solving the wrong problem. The process was never the point. If we grade writing by how little AI touched it, we are measuring the purity of the method, not the worth of the idea. It rewards effort and ignores whether there was any thought behind it.

What is left

If you cannot prove how something was made, the only thing left to judge is whether it is any good. Whether there is a real idea in it. Whether a real person, with a real stake in what he is writing about, stands behind it.

That is uncomfortable, because it is harder. It is much easier to check a box that says “100% human” than to ask whether someone actually has something to say.

The watercolor is safe from all of this, and that is probably why I hold on to it. It is the most uncorrelated thing I have - slow, on paper, no prompt, no model. And if you ask me for proof, I have it. Indisputable. No slop like Pangram getting into the mix.

I have no idea how to solve AI slop. But let me reframe it as three questions.

One, do we need another possibly biased AI black box to tell us something is slop?

Two, if we let go of proof - if we accept that we can no longer tell the human from the machine, and choose to judge only the idea - do we lose something real?

Three, do we really know what authenticity is? It has changed, numerous times, across the ages.

#Authenticity #Art #AI #Writing #Reflections

My days now are mostly training people on AI - with a startup being built stealthily on the side, and the odd bit of helping folks write things, which I genuinely enjoy.

And training people day after day, I keep seeing the same thing: plenty who've done some AI course, very few whose work actually looks any different.

That's the gap. We've mistaken doing a prompt-engineering course for fluency, and buying the newest tool for transformation.

So when the folks at Asia Tech Alliance invited me to co-author a piece on this, I said yes pretty quickly - it's something I care about. The paper pulls together some ideas on what to do, and frameworks for thinking about it.

Many from far greater minds than mine (especially the folks listed below). And one we synthesized: the Fluency–Transformation framework. Two dimensions - worker capability on one axis, organisational capability on the other - and four places a firm and its people can end up: Exposed, Frustrated, Extractive, or Transformed.

The premise is simple. Real transformation isn't measured by licences sold or courses completed, but by whether the work itself looks different twelve months on.

This piece is something dear to me, and I hope it helps folks think a little differently about AI at work. Even if it shifts the conversation just a bit, I'm happy.

So do take a read below, and tell me, and the folks at Asia Tech Alliance - what you think.

Ai Work

23.1MB ∙ PDF file

Download

Download

Or if you prefer a direct link - https://www.asiatechalliance.com/makingaiwork

Christina Schönleber | David Autor | Franziska Ohnsorge | Soon Joo Gog | Ingo Laubender | Pei Ying CHUA | Priyank Hirani | Trisha Suresh | Young Park | Lih Shiun Goh | Natasha Gašparič-Gray | Sharon Tan

#MakingAIWork #ArtificialIntelligence #FutureOfWork #SMEs #AsiaPacific

Adoption without understanding. Shallow literacy.

Prompt engineering as a career skill. Now loop engineering. Courses that teach you the latest tool in the most superficial way.

The hype has turned a genuinely interesting discipline into a noise machine. It’s bothered me for a while.

A busy week of delivering and building. The first run of my AI risk management course for the The Association of Banks in Singapore. A panel for private bankers. A few chats with founders asking if I’d like to advise or collaborate.

The steps below are some of my small answers to the hype.

Teaching the boring stuff

I enjoyed the first run of a deep dive into AI governance and risk management with the Association of Banks in Singapore. The content isn’t about how exciting AI is. It’s about AI risk management as a discipline and a system. Why AI fails, and what to do about it.

Everyone wants the excitement. I’ve been asked to excite an audience more than once. But what’s the point.

Get excited for what. So you can misunderstand what AI is and what it can do?

The course is my small answer to AI training that doesn’t build anything durable.

Art as a different kind of answer

A watercolour every night before bed. I illustrated for years before this, but watercolour is its own discipline, and not every piece works. Many end up in the bin. I really miss the undo button.

But the practice helps. The eye gets better. The hand gets more confident. The gap between what I see and what appears on paper gets slightly smaller.

Gradient descent, one piece at a time.

In a world where AI can generate infinite images in seconds, making something by hand at midnight grounds me. Slow. Imperfect. Irreducibly human. The progress doesn’t look like progress most nights. Just a brush in water. Pigment on paper.

My little act of rebellion in a world of hype.

AI assessment

Job AQ. Not about what tools you know, but how you think about AI - whether you understand it, can work with it, evaluate it, and own it.

Job AQ is a collaboration with . It’s our small answer to the broken world of AI training. Rather than fighting with the noise, we are coming at it from the other end: assessment rather than courses.

I’d be grateful if you tried it and told me what you think - jobaq.me.

None of these are big answers. The course is one course. The art still fails more than it succeeds. Job AQ is a 15 min exercise. But they’re what I can do.

#IndependentLife #AIRiskManagement #Art #JobAQ #Reflections

It’s been just but over 6 months since I left a full-time job at MAS. From two decades of a routine job to the uncertain world of a free agent. This question comes up quite often.

I have no idea how to answer such a question. Cause I have no idea what the term ‘happy’ actually means. Sometimes playing nice music already makes me happy. Or waking up to my wife next to me. Or even a Whatsapp chat with a good friend.

I think a better state to think about the shift is ‘clarity’.

And the past week of building and delivering has led to some thoughts on this. A webinar for insurers. A talk for Prudential. Slides for my upcoming deep dives into AI risk management for the Association of Banks in Singapore. Slides for an executive education programme for some Chinese banks at the end of July. A lunch with a collaborator on risk management for Agentic AI and my ex-colleagues. And pilot testing of the first product for a startup with a collaborator (not on AI risk at all). And lots of watercolors.

And when it comes to clarity, leaving a full time job is both the worst and best thing.

Why?

Worst & Best

Leaving a job is the worst for clarity because everything tied to that identity disappears. The title. The salary. The schedule. The structure. For someone like me who spent ~ two decades in the Monetary Authority of Singapore and the public service, it’s inevitable that my identity is tied to such a solid institution and the sense of mission there. Having finding yourself in a space with no clear role, no clear identity, and perhaps no clear idea of what you what is quite disorientating.

But it may also be the best for clarity. Because you find out very quickly what you actually need to reach out for when there is no one and nothing telling you what to do. To be clear, I’ve not reached a state of clarity yet, but I’m happy that I have found 3 things that help me get there.

Freedom

The first thing. Being free to say what I want.

2 decades in an institution trains you to be careful. To hedge. To represent a position rather than hold one. To speak on behalf of something larger than yourself. And unfortunately, that sometimes means saying a lot but meaning nothing.

I walked into the Prudential session this week and thought - no risk here. I’m going to say exactly what I want. And that has been my approach for the past few months.

So I did. That prompt engineering is not a skill and that one should not waste time on it. On the useless things in AI governance. That principles alone are just decoration. That the most important thing in AI is the boring stuff. The lesson here - don’t book me for a talk if you just want hedged points.

This is also what the weekly reflections have been. And many other posts and talks. The institutional filter, finally removed.

Building

The second thing. Building.

Not just asking questions. Or theorizing about things. Or meeting people.

Actually making things. Over the past 6 months, I have been trying to build my training practice - Quaintitative. Somewhere that I can deliver training that I believe in. It’s still not there yet. But I will persist.

But also not just that. Also a product that assesses AI competencies with a collaborator. And more things are coming - a platform to make AI and AI risk management more accessible. Follow on to find out more.

Building things makes me happy. And it helps to ground me.

Art

The third thing. Something surprising.

The problem with being able to say what you want, and to build what you want is the inability to switch off. With a full-time job, I don’t find it hard to detach. No matter the workload. It is after all, someone else’s demands.

But when it’s your own ideas, the mind never switches off.

Being an entrepreneur is different. You are never settled. The mind keeps going on after midnight. There is always something unfinished, something to build, something to worry about.

And I have found a solution. A watercolor every night before bed. Even when it doesn’t work. Even when the piece ends up in the bin. The brush in my hands, dipping into the water, mixing colors into gradients on paper. Letting the water flow and glow.

The art turns it off. One hour before bed where presence is required and everything else waits. You cannot paint and worry at the same time. Try it.

It also does something else. In a world where AI is eating everything - identity, creativity, and the bullshit is overwhelming - making something by hand is the most uncorrelated thing I have.

Slow. On paper. No prompt. No model. Just observing whatever is left of Singapore before it disappears.

There is no happy

People ask if I’m happy. I’m not sure what that is.

What I know is this. Six months in, I have no regrets. Not about leaving. Not about the plans that failed.

And I am happy that things are getting clearer.

#Clarity #IndependentLife #Art #AIRiskManagement #Transitions #Reflections

How to cope with change in AI governance.

I spent an hour with a room of senior folks from a major insurer earlier. My presentation was titled “Some thoughts on scaling AI governance”. Near the end, someone posed a question that’s on the minds of most folks. But with a new analogy.

In most of finance, he said, things are quite stable. First line takes the risk. Second line checks it. Third line audits it. Like positions on a football field. With AI, the field itself is shaking. The technology shifts. The regulatory standards shift. The models change every few months.

So how do you govern something that will not hold still?

Let me jot down how I answered it. And I wanted to connect my answer to the room to something that has been happening in research. Which I did not have time to go into there.

Be boring, don’t chase the AI model

My first instinct, as usual, was a boring one.

Whether the field is shaking is a matter of perspective. And many things still hold.

You do not anchor your governance to the thing that keeps changing. For AI, it’s the AI model, the LLM, the third party AI provider that does not tell you they made a change. You anchor it to the thing that is not moving.

And the task and the risk management system around the task does not move. Fraud detection is fraud detection. Insurance underwriting is insurance underwriting. The task defines what a good answer looks like. Not the AI model. And what a good answer looks like defines how you decide. None of that changes when the AI underneath changes.

Against the task, the model can be the swappable part. From the hopeless Llama 3 to the mythical Mythos - if you have a good system in place - e.g., a good evaluation and test set to measure what’s good, good processes and a solid platform to evaluate and test against, your endgame is the same. It does not matter that AI keeps changing.

The task remains the same. The system surrounding the task can also be the same.

In the room, I said this could be viewed as the equivalent of a harness for AI governance and risk management.

And folks have moved on from prompt to harness to loop engineering LLMs. The only one I think that has some substance is harness engineering. The rest are just hype. (Sidenote: The room also asked how to keep up in terms of skills sets. I said do not hire someone to train your stuff in prompt engineering because it’s a waste of time. Hire someone to train your staff about how to think with AI.)

And for the harness, we have had one in risk management far longer, we just never named it. It is the set of policies and processes, procedures, protocols, evaluations, testing sets, platforms, boundaries and controls we wrap around whatever we need to manage the risk of. Need not be a model. Could be a derivative. A new product. But the harness, if you have seen it across all these developments, can be remarkably stable.

And it compounds. Example, the more reps you get running evaluation and testing, and expanding your testing datasets, the faster you absorb each new model that lands - because you already know what you are testing it for.

Capability and risk are just different sides of the same coin

Separately, a bunch of papers this past quarter has been circling the exact same idea - from the other direction. Not how to govern a moving target, but how to squeeze more out of one. And in every one of them, the model is held completely still.

Meta-Harness searches for better scaffolding automatically, weights frozen. On a classification task it jumped in performance while using four times fewer tokens, better and cheaper (and probably faster too), without thinking about model. (arxiv.org/abs/2603.28052)

The Harness Effect did the same, but for cost. Across six models and twenty-two tasks, changing only the orchestration cut cost per task by about 30-60 percent at equal quality - and moved the bill more than the entire gap between the cheapest and most expensive model. (It is their own harness, and they are probably selling something here, so read with a pinch of salt.) (arxiv.org/abs/2607.06906)

From Model Scaling to System Scaling says the same. Once a model is good enough, the next gains come not from the mind inside the machine but from the system around it - how it remembers, what it retrieves, how it routes work, how it checks itself. The frontier, it argues, is that surrounding discipline, not the model. (arxiv.org/abs/2605.26112)

There are others. And there will be more.

Read together, they say one thing. Capability is becoming a property of the system, not just the model. The model will keep changing. But the harness is perhaps where the leverage now lives. Especially for everyone else but the frontier companies.

My point above on governance and risk management has no papers. But I don’t think it’s very different. Same coin, just different sides.

Nothing new under the sun

Finance has done this before. It always does.

Every so often a new instrument arrives that no one has priced - portfolio insurance, credit derivatives, structured products, the first trading algorithms. Each time, the instrument is new and the discipline around it is not. You size the exposure. You set the limits. You monitor. You keep the authority to stop. The thing in the middle keeps changing. What we wrap around it barely does. AI is just the newest thing in the middle.

Take structured derivatives. A bank invents a product with a structure no one has priced. There is almost never enough data to validate it, the data sources do not exist yet.

So what did they do? They did not refuse to sell it. They also did not pretend to a certainty they did not have. They deployed it inside boundaries. How much you can sell. To whom. Which thresholds you watch. When you pull the rug.

AI is the same shape of problem. You can train and test on the data you have. But until it hits real deployment, you do not truly know how it will behave. So you deploy it the way finance has always deployed the genuinely new, in a proper harness, with conditions, monitored, with a way to stop it.

The harness is what holds those conditions. The model is just what is sitting inside it this month.

First principles, not the method of the month

The trap is to govern the model by its techniques. Because the techniques churn fastest of all.

I get asked constantly: there is no settled explainability method for generative AI, none for agents, so how do I use one for underwriting or fraud?

Here is the reframe. Explainability is not about explainability. It is about understanding. You do not need to track whether the method of the moment is SHAP or some mechanistic-interpretability paper from last week. You need to know what you want the understanding for - who has to understand what, to make which decision. Get that right, and a good evaluation set will often teach you more about the system than any named method would.

Same for everything else. You do not memorise every metric. You ask what it measures, and whether that is the thing the task actually cares about.

This is also what the harness is made of. The methods - the explainability technique, the eval metric, the guardrail of the week - are just parts you slot in and swap out. The harness is the frame that decides what each part is for.

Ask from first principles and the churn stops mattering, because the questions underneath are stable even when the methods on top are not.

And no - please do not go buy another prompt-engineering course. That is not the skill. The skill is learnable, and it is not that hard. How do models actually work. Where are their edges. What can they not do. If someone says they want a language model to forecast markets, you should be able to say: it is trained on text, show me the evals, show me what it beats. Don’t bullshit me. That question does not need a technical deep dive. It needs a mental frame.

But the danger moved too

I would not read those papers and only feel reassured, though. If the harness is where the capability now lives, the harness is also where the failures now hide.

A stale note in the model’s memory is more dangerous than no note - it hands the agent confidence at the exact moment it should stop and check. A helper sub-agent can fail quietly, because its answer sounds plausible and nothing downstream checks whether it is true. Put a model in charge of watching another model, and you have grown a second thing that can be wrong.

Scale the harness without scaling your ability to see into it, to stop it, and to answer for it - and all you have built is a greater risk surface.

So, the shaking field

The model will keep changing. So stop nailing your governance to it.

Build the harness - the durable set of controls, checks, tests and boundaries around whatever model is inside it this month - and the shifting technology becomes something you swap through, not something that swaps you out. The engineers scale the harness to get more out of AI. We scale it to stay in control of it.

I do not have this fully worked out - if I did I would be rich, and I would be selling it. But this is how I am starting to think about it.

#AgenticAI #AIRiskManagement #AIGovernance #AISafety

When I was young, I watched my uncle work on his bonsai. Some branches he cut off. Others he wrapped in wire and bent, slowly, until they grew the way he wanted.

Two conversations in the past week have made me think about the connection between his craft and Agentic AI governance.

The first. A comment left by Georg Zoeller on my post. On how constraining agentic AI with rule-based guardrails would not work. His observation was sharp, and justified. If we can pin an agent down with rules, why use AI at all? Why not just write the rule?

My reply was that all models (not only AI) are just mapping functions. Inputs to outputs. Rules too.

When we know what we want, we should certainly use a rule to do the mapping. When we do not know, and we want the machine to learn the pattern that maps input to output, we use AI. The role of rule-based guardrails, then, is not to replace the AI with a rule. It is to reduce the space of patterns AI is allowed to traverse. They prune off the patterns we do not want, and constrain the ones we do.

The “hope” is that the envelope of possible behaviour left after such pruning can be trusted.

The second conversation was during a lunch just now with Lukasz Szpruch and some ex-MAS colleagues. The question was about how far rules could go. Can we realistically have rules for everything in an agentic system? What is the actual role of an LLM as a judge? And where does old-fashioned evaluation and testing fit?

It left me thinking. So let me jot down my quick thoughts.

The tree

Picture an agentic system as a tree. Unlimited branches, each a path the agent might travel. If you let it be, it gets quite unruly.

Not all paths are equal.

In terms of impact - some are truly bad - the ones that cause real, irreversible harm. Some are potentially bad, but fine if controlled. Many are simply not great - low stakes, reversible, the kind you can afford to get wrong now and then. And some are exactly what we want.

In terms of practical controls - some actions can truly be constrained with rules, e.g., a recency check, a sources whitelist. Some may require great flexibility, e.g., filters of open-ended advice or answers to questions. And many in-between.

Take this example - an agent that resolves customer billing disputes. It reads the ticket, looks up the account, decides whether a refund is owed, issues it, replies to the customer, and closes the case.

Even that small workflow branches enormously.

Some branches are truly bad: refunding to an account that is not the customer’s, issuing a refund far larger than the charge, exporting the full transaction history, switching off the audit log.

Some are potentially bad but manageable: a large refund is fine, if a human signs off first.

Many are low stakes and reversible: the wording of the reply, which help article to cite, whether to offer a small goodwill credit.

And one narrow set is what we actually want: the dispute resolved within policy, logged, the customer treated fairly.

This is what changes when we move from a normal model to an agent. A normal model gives you an output. One thing to check. An agent chooses an action, then another, then another - and each choice opens more. The paths branch faster than anyone can list them. That explosion is the risk. Not a bad answer, but a bad route to a fine-looking one.

Better to fail loudly, than silently.

Cut, and wire

But you cannot check every path.

So I’m thinking if we can do what my uncle did. You cut, and you wire.

Cut - You cut the paths that must never happen. The truly bad ones. Delete the database. Move money with no sign-off. Send the customer list outside. You do not manage these, or monitor them, or hope. You remove the branch, so the agent can never walk down it. Zero trust.

Everything that is left, there could be three kinds of wire.

Rigid wire - rules. Stiff, certain, holds its shape on its own. Where a path can be constrained by a plain rule - refunds under a set amount only, always to the original card, stop and ask above a threshold - use it. A rule adds no new paths of its own. It is the strongest wire you can use, and the cheapest to trust. Implement as many of them as you can find.

Supple wire - a model as judge. Some paths are too fuzzy for a rule. Whether a complaint is genuine. Whether a reply is rude. There you let an LLM, or a smaller model (ML or SLM), watch and bend the branch. It is far more flexible. But this wire has a mind of its own. The judge is itself a model, with its own paths, its own ways to be wrong. You gain judgement, and you grow a second tree beside the first. So use it deliberately, and govern it as carefully as the thing it guards. Still better than leaving it to a system prompt and a prayer.

Your own hands - evaluation and testing. Some paths that you cannot trust to either wire. Those you bend yourself. You sit with the agent, predict where it will go, probe the branches and edges, and correct by hand. It is the most flexible control there is, and the least scalable - you only have two hands. It is also how you learn where the other two wires need to go. Every test that surprises you is a new rule waiting to be written, or a judge waiting to be posted.

Which branch gets which wire

Three wires. Different trade-offs.

Rules are certain but rigid.

Judges are flexible but grow their own paths.

Your hands working on evaluation and testing are wisest but do not scale.

And the order should perhaps be this.

Push as much as you can onto rules, because they are certain and add nothing. What rules cannot hold, wire with model judges, and accept that you have added paths in exchange for flexibility. What neither can hold, keep in your own hands, and let your evaluation and testing tell you what to promote to a rule next season.

Every wire is a way of narrowing the envelope of patterns the agent can express. A rule narrows it hard. A model judge narrows it softly. Your hands narrow it one case at a time, and you get better at it with practice. None of them turn the AI back into a rule. They only decide how much room it gets.

None of it kills the tree. My uncle’s bonsai lived for decades.

Is it so for Agentic AI? Not sure. But this is how I am starting to think about it.

#AgenticAI #AIRiskManagement #AIGovernance #AISafety

I wrote about gradient descent three months ago on LinkedIn. I was figuring out how to wander the uneven terrain of independent life. The failures. The plans that drifted. Finding the right angle to navigate.

I’ve learned to be comfortable with the uncertainties and disappointments of this new phase. Even relaxed enough to pick up the brush again.

But somehow I met gradient descent again. This time when trying to learn to learn art again.

A light week of meetings. A bunch of regulators from around the world I am mentoring for a course on AI in financial services. A collaborator on the Agentic MRM paper who was in town. A consultant thinking of moving to a senior AI role in a bank, asking for my views. An ex-colleague from my strategic planning days, now in communications. A senior risk manager between jobs, developing scented cleaning supplies as a hustle.

Lots of building otherwise. And art. I’ve built up the discipline of doing watercolors every night before bed. Which started another round of gradient descent. And a bit of angst.

The Gap

A retrospective I wrote last week made the art sound resolved. An interest that could be an uncorrelated asset in this world of exhausting AI hype. (Side-note: I’m starting to think I have a love-hate relationship with AI. I’ve always loved the discipline, but it has increasingly become populated by hypesters and scammers that rile me.)

I thought I had hit the ground running with my watercolors. The practice had returned. A satisfying piece here. A passable piece there. Enough momentum to feel like my old friend was back. I thought the next five pieces would be even better.

The next five pieces across the week made me realize it had not. Almost none of them worked. My old friend had not forgiven the gap.

Gradient descent is how a model learns. It makes a prediction. Measures the gap between the prediction and the right answer. Then adjusts in the direction that reduces the gap. Then repeats.

It’s called gradient descent because of how it finds that direction. Imagine a hilly landscape. You’re standing somewhere on it, blindfolded. You want to get to the lowest point - the valley. You can’t see the whole landscape. But you can feel the slope beneath your feet. Gradient descent says - follow the slope downhill. One small step at a time. Measure the ground. Adjust. Step again. Eventually you reach the bottom. Or at least somewhere lower and better than where you started.

The gradient is the slope. The descent is following it.

Three months ago I wrote about gradient descent in independent life. Misses. Proposals that went nowhere. Plans that drifted. All of it telling me the slope and which way to go next. Lots of uncertainty. But I got reasonably comfortable with that. Not because it got easier. Just because I could detach from the results. And go with the flow.

Gradient descent in learning to learn art seems different.

Somehow, I find it harder to detach from the gap. The gap between what I see in my head and that muddy mess that appeared on paper in front of me.

And there’s the obsession and ambition problem. When you restart a practice you start looking at the masters. Chung Chien Wei and his effortless wet-on-wet scenes. Thomas Schaller’s architectural washes, luminous and controlled. Alvaro Castagnet’s explosive loose strokes that somehow resolve into something complete.

You know what good looks like. You can feel the gap between that and what’s on your desk. And the gap isn’t abstract. It’s a piece you spent an hour on that looks nothing like what you intended.

One piece almost hit the spot. For a moment something was clicking. Then I lost it. The next piece drifted back toward the same problems as before.

Getting close and losing it is worse than consistent misses. When nothing works you can tell yourself you’re still finding the slope. When something almost works you know exactly what you’re reaching for. But then it becomes elusive again.

The Way Forward

The difference between knowing gradient descent and feeling it is considerable. I wrote about it in the context of independent life.

What I didn’t write about is what it actually feels like to sit with a miss. To look at something that isn’t working and try to stay patient. To not force it. To trust that the signal is in there somewhere even when you can’t see it yet. And to let go if it really just needs to go.

Three months ago the gradient descent was in independent life. Now it’s in a practice. More immediate. More personal. More uncomfortable in a different way. But not very different.

And I think the way forward is simple.

Pick up the brush again tomorrow. That’s all gradient descent really requires.

#GradientDescent #Art #IndependentLife #AIRiskManagement #Transitions #Reflections

Don’t force it. Let things flow.

I had scribbled these words on a post-it. I was two months from leaving MAS. I think it was a salve for my anxiety.

For most of the six months since I left, I don’t think I listened to my own note. I forced plenty. Endless coffees, lunches, talks, plans.

Recently, I’ve started to get back to these six words. And last week, for the first time in more than a year, I managed to dive deep into watercolours again.

A good friend at work had once remarked that seeing me do art was a good indication of how bored I was at work. I’ve come to the realisation that it’s a bit more than that.

And since this is the six-month mark, it’s a good time for reflections on some personal lessons. And how my view of AI has shifted. Which could perhaps be useful for some others.

My lessons

The first, and the main one. Run experiments and reflect on them.

I left MAS after eighteen years and walked into a void. I did not search for another job before leaving. Even the plan I did write down at the end of 2025 never happened.

In the end, the last 6 months became a series of experiments.

More than two hundred conversations across twenty-five weeks. At least two to three talks or panels every month. Old friends and complete strangers. CROs and PhD students. An anthropologist studying money, and a CEO I mistook for a fellow trainer. Collaborations across the world.

Every week I reflected on what I learned from these conversations. Some of it was connected to AI. Some of it was just about life. People ask how I find the connections between these conversations. I read The Book of Ichigo Ichie recently. It says “notice coincidences” as one of the ten rules of ichigo ichie. I guess this is what it means.

In the age of AI, I think this is how learning looks like.

Two, the filter beats the plan.

My original plan was 80% wrong. My filter - interesting work, people I like, freedom, the work done properly - was almost always right. A plan imposes a conclusion and makes you bend everything to fit it. Even if it feels wrong. A filter just tells you what to walk away from.

Build a filter early, even before you can fully articulate it.

Between a plan and a filter, I’ll take the filter every time. They’ll help you avoid some early missteps.

Three. You have to actually cross.

Not observe. Cross. Watching other worlds from a safe distance is not enough. The value is in gate-crashing them, awkwardly if one must. The insurance world, where I mistook a CEO for a fellow trainer. Legal AI. Digital assets. An MBA classroom on the other side of the world. US AI governance. African bank and insurance boards. Healthcare workers learning deep learning. Wealth managers and family offices. A central-banking podcast. And many more.

I am an introvert who loses energy when I meet too many people. A side-effect of these crossings - I’m feeling at ease now in crowds. A simple rule is all it takes. If I feel like chatting with someone, I do it. If not, I walk away.

Four. Flow is just gradient descent.

Misses are information. The proposals that die, the rooms that don’t want your lines, the conversations that trail off - they feel like failures, but they point you in a better direction.

In gradient descent a wrong step isn’t failure; it’s the signal that tells you the slope, and which way to step next. You cannot know which direction is right without the wrong ones. Expect a lot of them early, and don’t mistake them for verdicts.

Five. Guard the freedom you left for.

If one leaves for some kind of freedom, protect it like the scarce thing it is.

That means cleaner no’s - to the gig that pays but owns your calendar, to the room that drains you, to the prestige that quietly puts you back in a box.

It means letting no single party own you, and having no single point of failure. And it means keeping at least one thing in your life that owes nothing to the work.

If the freedom is the whole point, don’t trade it back the first time saying no feels awkward.

How my view of AI shifted

I started by seeing the unevenness.

AI was in every single conversation, but understanding was an uneven valley - someone shipping AI products daily, someone else whose work was untouched, same city, same week. I stopped seeing one AI race and started seeing dozens, on different tracks, most runners unable to see each other.

Awareness had stopped being the solution and become the problem, setting expectations nobody could square with reality.

While some folks were going from prompt and context engineering to harness and loop engineering, more sane folks were wondering how an LLM that made a silly hallucination one in 20 times could be trusted to perform an action that mattered.

The boring fundamentals matter more than the hype.

When sixteen agents built a working C compiler, what made it work wasn’t the frontier. It was the boring list - tests, documentation, monitoring - the same list from the risk guidelines I wrote.

Real understanding compounds and builds a flywheel. Prompt engineering resets with every model update. AI courses that just teach one to prompt and pray are not building AI literacy but just scams in disguise.

The scaffolding is important but thinning.

The same week a few of us published a paper extending model risk discipline into the agentic world, SR 11-7 - the standard I’d been referring to for more than a decade - was quietly retired.

The real rules seem to be thinning out just as the systems get more complex. We have plenty of publications on responsible AI. But tell me, how many bite and are effective?

So I guess we just have to fall back on first principles, not frameworks. And I learned to distrust fluent reasoning, in models and in myself.

Coherence is not correctness.

And I ended more sceptical than I began.

Not of the technology - of the expectations around it. I think they are overblown. Whether AI becomes as common as electricity or quietly stalls, an identity built entirely on it is fragile either way. That is what sent me back to water and paper.

What I do now

The shape that emerged is not the one I planned.

Training for a few associations and institutes. A few adjunct roles. Some writing gigs. A startup I’m building with a collaborator. A few advisory gigs on the horizon.

No single party owns me. No exclusivity. If one fails, no big deal. I can still survive.

Before I start any gigs these days, I know to zoom in on the non-solicitation, non-competes, indemnity and intellectual property clauses. These are the ones that can matter more than the money.

I’ve stopped saying yes quickly. The no’s have gotten cleaner. I show up to speak when I have something to say, and otherwise I’d rather listen.

And I’ve started hedging the one risk I couldn’t diversify away. Everything I do sits in the same asset class - AI. So the uncorrelated asset is the art. By hand. Slow. On paper. No prompt. No FOMO. Just observation and whatever is left of Singapore before it disappears.

It may not take off. The world of art is brutal. But it’s irreducibly human.

Gradient descent has one catch. You follow the slope down, you correct, you flow, and you arrive somewhere that looks exactly right. But it might only be a local minimum. A valley within a valley. A few more steps might have taken you somewhere better, and you’d never know.

So I won’t pretend six months converged on an answer. It’s just a start.

The twenty-four weeks

Every Monday for six months, I tied that week to an AI concept. I stopped numbering them around week twenty - but here they are, in order. Start anywhere.

(Each links to the original LinkedIn post.)

Week 1 · 12 Jan 2026 — Valleys, Shifting Sands & Pillars in AI
AI everywhere, understanding nowhere near even. The uneven landscape.

Week 2 · 19 Jan 2026 — The Darn Messy Middle of AI
When awareness stops being the solution and becomes the problem.

Week 3 · 26 Jan 2026 — Phase Shifts in AI and Life
Not pivots. Changes of state. Why I’m optimistic about AI.

Week 4 · 2 Feb 2026 — Resonance - AI at Work and in the Classroom
Becoming a LinkedIn Top Voice the same week I rediscovered my voice.

Week 5 · 9 Feb 2026 — The Flywheel
Sixteen agents, one C compiler, and the boring fundamentals that made it work.

Week 6 · 16 Feb 2026 — Many Different Worlds
There isn’t one AI race. There are dozens, and most runners can’t see each other.

Week 7 · 23 Feb 2026 — The Outsider
Transfer learning, and why nothing we learn is wasted.

Week 8 · 2 Mar 2026 — Alignment
The boat-race agent that spun in circles, and freedom as non-negotiable.

Week 9 · 9 Mar 2026 — Context
The context window, finite attention, and learning to filter.

Week 10 · 16 Mar 2026 — The Edge of Action
Three gaps, ten weeks of evidence, and what I finally decided to build.

Week 11 · 23 Mar 2026 — Grounding
The symbol grounding problem, and meeting rooms where they actually are.

Week 12 · 30 Mar 2026 — Diffusion
Graph diffusion, signal versus noise, and the cost of saying yes too much.

Week 13 · 6 Apr 2026 — Gradient Descent
Why the misses weren’t failures. They were the gradient.

Week 14 · 13 Apr 2026 — Unfinished
No prepared deck survived the room. And fallow isn’t drought.

Week 15 · 20 Apr 2026 — Irony
SR 11-7 retired the same week we argued for more discipline, not less.

Week 16 · 27 Apr 2026 — Thinking Out Loud
Chain-of-thought can be confident, fluent bullshit. So can I.

Week 17 · 4 May 2026 — Serendipity and Emergence
Rojak, introversion, and building the conditions for emergence.

Week 18 · 11 May 2026 — Strange Attractors Again
Chaos isn’t random. It’s shaped by topology.

Week 19 · 18 May 2026 — Impermanence and Drift
Concept drift, disappearing spaces, and a plan turned unrecognisable.

Week 20 · 25 May 2026 — The Crossings
The value isn’t in observing the worlds. It’s in actually crossing.

Week 21 · 1 Jun 2026 — Sum of the Parts
Skill-stacking, freedom of choice, and a hard question about privilege.

Week 22 · 8 Jun 2026 — The Dream and the Plan
Breaking from the Plan is the beginning, not the answer.

Week 23 · 15 Jun 2026 — 5 Useless Things
Useless to box, to force, to perform, to accommodate, to follow convention.

Week 24 · 22 Jun 2026 — The Riskiest Play
Concentration risk, an uncorrelated asset, and picking up the brush again.

I write a lot on both LinkedIn at https://www.linkedin.com/in/garyang/ and also Simply Boring AI, my Substack at simplyboring.ai. Sometimes I cross post. Sometimes only on one. I’m still figuring out works best. So follow me on both.

And if you’d like to work together - workshops, teaching, advisory - I’m at quaintitative.com.

#IndependentLife #AIRiskManagement #Transitions #Reflections #GradientDescent

The AI risk management guidelines I wrote for Singapore’s financial sector grew out of both, and they rest on three units of governance: the model, the system, and the use case. Lately something seems missing.

These three units - the model, the system, and the use case - are the ones we already know how to govern, but they share a weakness. Each is specific, tied to one particular application.

Above them sits a second set of three that we are beginning to think about for governance: the archetype, the capability, and the workflow. These seem to be better as they generalize.

Why is this important? A unit of governance is something you point a control at. Too granular and it may work, but not scale.

Let’s draw a line between 6 units. The units below the line are where governance lives today. The units above it may be worth considering for governing AI at scale.

The six of them, and the line between, look like this.

Below the line

The three units below the line are not new. We imported each of them from a discipline that already existed - model risk management, technology risk management, and the use-case registers that AI governance frameworks already lean on. That is their strength and their limit. The methods are mature. But they were built for a world where each thing you governed was specific and countable.

The model

I’ll start where I always do, with the model. It is my favourite, and it is the one thing that makes an AI system different from any other piece of software. The first models I met were simple curves - bootstrapping zero rates off swap quotes in a financial engineering classroom, then their more sophisticated cousins, Black-Scholes, SABR, Hull-White, Heston.

For years a single supervisory letter, SR 11-7, was the only guidance that governed any of it. I took a Masters in Financial Engineering but was not a great student, and so I stayed confused about those models in finance for a long time.

What finally made them click was meeting their successors - decision trees, random forests, gradient boosted trees, then the deep learning models, the RNNs and CNNs, and eventually LLMs. They are all more general versions of the same idea. Even an agent is just a system wrapped around an LLM.

Once you can see the parts every model shares - the function you choose, the objective, the optimization method, the way you evaluate it - the questions you need to ask barely change from one to the next. They all come back to those parts, and to the assumptions beneath them.

But something did change on the way from the curve to the LLM, and it matters for governance. A Black-Scholes model is written down. You can read its assumptions off the page and argue with them. A learned model is not written down in that sense - its behaviour lives in weights shaped by data, not in an equation you can inspect. So validation has to change. It stops being a check of the maths and becomes a probe of the behaviour. You can no longer read the model; you can only interrogate it.

That is the real break, and it is why so much traditional model risk practice feels half-applicable to AI. The questions survive. How you answer them does not. And we see this in Agentic AI increasingly due to their ability to take actions and not just generate outputs.

But that shared structure is also where we see commonalities in risks. I think of them as three U’s.

• Uncertainty - the irreducible randomness you can never remove, and the reducible gaps you can close with more data. Every model has both. The work is knowing which is which, because you waste effort trying to engineer away the part that won’t move.

• Unexpectedness - the way more complex models behave in ways nobody designed, in emergent capabilities, adversarial weaknesses, and misalignment. The more general the model, the more of its behaviour was never specified by anyone, which means more of it is found rather than built.

• Unexplainability - the varying degree to which we can actually say why a model decided what it did. Transparency, explainability, interpretability are all really one question asked at different depths: can we understand this enough to stand behind it? Understand. Not quote some figures generated by an explainability method you don’t understand.

Governing the model, then, is mostly documentation, independent validation, and benchmarking - the same spine as classical model risk. The trap is treating the benchmark as the answer. A model can top every public benchmark and still fail in your context, on your data, against your adversaries, because the benchmark measured a general skill and you are deploying a specific one. The benchmark tells you the model is capable. It does not tell you it is safe here.

It is telling that when the Fed, the OCC and the FDIC replaced SR 11-7 this April with SR 26-2, after fifteen years, they carved generative and agentic models out of scope as too novel and fast-moving. I understand the caution. But the effect is that the fastest-moving, least-understood models are the ones left with the least guidance. Given how tightly the quantitative and the modern connect, that exclusion still puzzles me.

The system

A model, though, never runs on its own. Wrap code, tools and memory around it and you have a system - and for most of my career the system was someone else’s problem. I came up in model risk: development documentation, validation reports, stress tests. Technology risk lived down the corridor, a different team with different acronyms, and I treated it as none of my business. AI changed that.

To see why, it helps to name what a system actually is. Around the model sits the prompt scaffolding that frames every request, the tools it is allowed to call, the memory it carries between turns, the retrieval that decides what context it sees, and the guardrails bolted on at the edges. Every one of those is a knob. Turn any of them and the same model produces different behaviour. The same prompt, through two different stacks, lands in two different embedding spaces and comes back with two different answers. So a model card tells you very little on its own. Two deployments of the same weights are two different risk objects, and the thing you actually have to govern is the configuration, not the checkpoint. We call them fancy names like harness or loop engineering these days for Agentic AI. But it’s really just a system with slightly more dynamism and autonomy.

This is where most so-called AI incidents really live. The system around the model is increasingly where the performance, the variance, and most of the failure modes actually live, and it cuts both ways. On the upside, changing nothing but the harness around a model can lift its score on the same benchmark dramatically, sometimes close to double - the model didn’t get smarter, the system around it got better at using it. On the downside, the system is where the incidents are landing - remote code execution in the Model Context Protocol, supply-chain compromises spreading through AI tooling, an over-permissioned tool, an unsanitized input, a secret written to a log. In almost none of them did the model fail. The thing around it did. We spent a great deal of worry on model monoculture. The system deserves more of that attention.

The use case

If the model and the system are how AI gets built, the use case is what it gets built for - and it is the layer almost every governance framework reaches for first. It is also the question I am always asked: what are the use cases? I find it frustrating because no one wants to understand the underlying mechanics and connect it to their tasks, but just want an AI use case served on a platter.

The pull is easy to understand. The use case is the unit the business names, the unit that goes into a register, the unit a board can hold in its head. It is the unit of value. So it becomes the unit of the inventory, and the inventory becomes the comfortable thing we can look at and boast about - “We have 1000 AI use cases!”

That comfort is the problem. A register of use cases gives you the feeling of coverage while the real risk objects sit a layer down, uncounted. It is the unit I trust least, for three reasons.

Not deep enough. A single fraud detection use case can hide transaction monitoring, an anomaly model, a generative assistant and a retrieval system underneath, each with its own failure modes. The use case enters the inventory once. Ten real things go ungoverned.

Not specific enough. Two banks can list the same wealth-advisory use case and mean completely different workflows, models, and risks. The label matches. Nothing beneath it does. So a control written against the label protects neither of them well.

Not scalable enough. Twenty use cases you can govern one by one. At two hundred the patterns underneath them simply repeat - retrieval, extraction, summarization, classification, recommendation, anomaly detection - and governing at the use-case level means redoing the same work every time you meet the same pattern wearing a new name.

The use case is the right unit to decide whether to build. It is the wrong unit to decide how to control. The unit that actually scales is one layer further down, or one layer up.

Above the line

Everything below the line is specific by nature, which is why the work never compounds - each new system is a fresh job. The units above the line are different in kind. They are patterns, not instances. Govern a pattern once and every instance that fits it inherits the work. That is the whole reason to climb.

The archetype

Go up a layer and you reach the archetype - the first unit that scales. I saw it during the generative-AI wave, before agents arrived: an LLM is general-purpose, but people kept getting more out of it by putting it in a box. Retrieval-augmented generation. Summarization. Classification. Information extraction. Code generation. Chat.

A general-purpose tool is almost ungovernable precisely because it can do anything - there is no fixed set of risks to reason about. The archetype trades a little of that generality for something you can actually hold: a known shape, with known failure modes.

Naming the box helps twice over. For building, it brings the recipe with it - the reference architecture, the prompt patterns, the libraries and settings that already work - so a new project starts from a known shape instead of a blank page. For governance, it brings the risk package with it - the failure modes that apply, the evaluation and testing metrics that matter, the controls you need - none of it worked out fresh each time. This is the clearest answer to the worry that risk management blocks innovation: done at the right unit, it is exactly what lets you reuse.

Governance stops being per-project checklist bureaucracy and becomes a library. You write the controls for retrieval once, and every retrieval system in the firm draws from them.

The same idea shows up across the common archetypes:

• Summarization - the dominant risk is the model inventing detail that was never in the source, so faithfulness is the metric you cannot skip, and high-stakes summaries earn a human check.

• Code generation - the risk shifts to security, so the check is correctness plus a vulnerability scan, with the small mercy that bad code tends to fail loudly rather than silently.

• Retrieval - the risk lives in what gets retrieved, wrong chunks or leaked embeddings, so you test whether the right context was pulled, whether the answer is grounded in it, and you watch the vector store closely.

Three archetypes, three different shapes of risk, but within each one the work is done once and inherited by every use case that adopts it. Constraints, it turns out, are what make a thing governable at all.

The capability

Archetypes tamed generative AI. Agents need the same move, one level harder, and it comes down to two questions: what can the agent do, and how far can it go on its own?

The first question is capability. In my mind, a capability is not what the underlying model can do, or how it scores on a benchmark; it is what an agent can actually do in the world, on real tasks - a repeatable building block with three things attached to it. The actions it covers. The authority it is granted to take them. And the evidence it must produce that it did so within bounds. Retrieve a customer’s transaction history is a capability. Move money between accounts is another, with far more authority and a far higher bar of evidence.

Defined this way, an agent stops being a mysterious opaque whole and becomes a set of capabilities you can reason about one at a time - decompose the agent into the capabilities it exercises, and compose your oversight back up from there.

This is why the benchmark misleads. A model topping a coding benchmark tells you nothing about whether an agent built on it should be allowed to merge to the main branch unattended. The benchmark measures skill in the abstract. Capability is about authority in the world. The two come apart, and governance lives in the gap.

Two pieces of work from very different directions arrive at the same place: GovTech’s Agentic Risk and Capability framework (https://arxiv.org/abs/2512.22211), by Shaun Khoo, Roy Ka-Wei Lee and Jessica Foo, and the work on runtime governance in finance (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6567199) that Łukasz Szpruch, Agus Sudjianto, Tanveer Bhatti and I did. Both land on capability as the better unit for governing agents.

And it scales the way the archetype does, but with guards baked in. Four agentic systems at a wealth firm - a service bot, a trade drafter, a compliance checker, a KYC assembler - look like four separate jobs, but they lean on the same handful of capabilities: document retrieval, fact extraction, policy checking. Govern those once - the authority each is granted, the evidence each must leave behind - and the rest is configuration, not construction. The fifth, sixth, seventh and even the hundredth systems cost almost nothing to govern, because they are new arrangements of capabilities you have already controlled.

The workflow

The second question - how far an agent can go - is the workflow. The workflow is what you allow it to do, designed up front: the branches, the approval gates, the points where it must stop and ask, or abstain. The trajectory is what it actually did, visible only afterwards. Most of the time the two match. The risk lives in the times they don’t, and runtime governance is the layer that watches the live trajectory against the allowed workflow and acts when they diverge - pausing for approval, refusing an action, stopping a run before it compounds.

Singapore’s Model AI Governance Framework for agentic AI (https://www.imda.gov.sg/-/media/imda/files/about/emerging-tech-and-research/artificial-intelligence/mgf-for-agentic-ai.pdf) frames the same thing as action space and autonomy - what the agent is allowed to touch, and how far it goes alone. They are two dials. Widen the action space and the agent can reach more of the world. Loosen the autonomy and it asks permission less often. Turn both up and the set of possible trajectories explodes, far faster than anyone’s intuition tracks. That is how almost every agentic incident happens: the workflow quietly permitted an action it shouldn’t have, the autonomy let the agent take it without asking, and nothing was watching while it ran.

The simplest test I know is whether someone can draw your agent’s workflow on a whiteboard - the branches, the gates, the places it must stop. If they can’t, then no one has actually drawn its limits, which means the limits don’t exist yet. And the trajectories, when they arrive, will not be the pleasant kind of surprise.

The whole picture

So there are six units, not three. Three below the line - model, system, use case - the established disciplines, and the specific. Three above it - archetype, capability, workflow - the ones we are still working out, and the ones that scale. The move up the line is a move from governing things to governing patterns, and it is the only version of AI governance I can see that keeps pace as the number of systems climbs.

I am still thinking this through. The line may move, and new units may yet appear above it.

But the question worth thinking about is a ridiculously simple one. Of the six, which is your AI actually governed at - and is it the one that can keep up as the number of AI systems grows? And it will.

This began as a series of short notes, written over a few weeks while I worked the idea out in public. Here it is in one piece.

This is the third and final part of my conversation with Johnny Lau.

Part 1, “My Shifu“ was how I met him, and his belief that technology forces artistic purity rather than threatening it.

Part 2, “Garfield’s Eyes“ was about separating the concept from the production. This part starts with the question - will AI replace us, and then goes to the quieter question underneath - how do you live alongside something like this?

Two Fears

Watch how people talk about AI and work, and almost everyone is running on one of two fears.

The first is the fear of losing out. Will I be automated? It’s a reasonable fear. One CEO recently described it as replacing “lower-value human capital”. The fear in the air is palpable.

The second fear is the opposite. It doesn’t feel like fear. It feels like ambition. Everyone’s using AI. If I don’t adopt it now, I’ll be left behind. So people automate before they understand, buy tools they don’t need, and call it transformation.

Fear of losing out. Fear of missing out.

In Singapore we already have a word for the first one. Kiasu - Singlish for the anxiety of losing out - is the national neurosis Johnny Lau turned into a character that sold over half a million copies. What I didn’t know, until this conversation, is that he also drew the second one.

I’d asked Johnny whether AI could ever generate a real Mr Kiasu strip.

“An AI produced Mr Kiasu strip will show Mr Kiasu and Ah Kow, his pet dog in different scenarios and it will be funny. But it is unlikely to create an Ang Mo character who calls himself Mr FOMO, and competes with Mr Kiasu for Ai Swee’s love.”

Mr FOMO. A Westerner whose neurosis isn’t losing out but missing out, competing with Singapore’s most kiasu character for the girl. It’s a cultural invention. The joke is in the gap between kiasu and FOMO. I realized Johnny had done something stranger than draw a character. He’d named both of the fears now running the AI conversation and made them chase the same girl. Two anxieties, one love triangle. Both running on fear. Neither of them living.

Co-Exist with the Storm

So I asked him the real question. Is AI just another wave? He’d lived through the comic boom, dot-com, mobile, animation outsourcing. Was this one different?

“AI is a climatic shift that is going to fundamentally alter the way we think and live. If the internet era represents a tsunami, then we are smacked right in the middle of a perfect storm. Almost no one will be spared, and it’s only a matter of how many of us can co-exist with the storm.”

Not survive the storm. Not beat it, and not ride it either. We should be thinking about how to co-exist with it.

That’s a different posture from both fears. Mr Kiasu wants to outlast the storm by building a wall. Mr FOMO wants to outrun it by chasing it. Johnny is proposing you live inside it - that you stop treating it as an enemy or a prize and start treating it as weather you’ll be making things in for the rest of your life.

Here’s how he does it. Three moves.

One: Stay Fluid

“Part of the solution is to stay fluid and go with the flow because machines does not comprehend ‘going with the flow’ as easy as we do. It can fake going with the flow, but it will never get it.”

AI learns patterns. That’s the whole of what it does. So the moment you organise yourself into a recognisable pattern - the same task, the same output, the same way every time - you become trainable. “Lower-value human capital” is just the corporate name for work that had become a pattern.

The kiasu instinct is to defend that pattern harder, to prove your worth by doing the routine thing faster. It’s exactly backwards. Staying fluid means refusing to be the thing the machine can most easily learn.

Two: Orchestrate, Don’t Just Execute

“It will take something for AI to mimic the way I had orchestrated a project. The orchestration of an idea or a series of ideas; the people coming together as collaborators; outcome that will have certain level of social impact.”

This is the answer to the FOMO fear. You don’t beat missing out by adopting everything. You beat it by getting clear on what only you can do - and for Johnny, that was never the drawing. It was the convening. Deciding which ideas were worth pursuing, putting the right people in a room, choosing what impact even looked like.

AI can produce the strip. It can’t run a creative enterprise. The machine executes. The human orchestrates. Living with AI means moving up the stack - from doing the task to deciding which tasks matter.

Three: Don’t Argue with Time

I’d argued that a character rooted in a Singaporean word might be permanently untranslatable - that kiasuism was a moat. Johnny didn’t buy it:

“Don’t argue with time if the matter is a matter of time, especially dealing with AI. To AI, there is no such thing as a Western or an Eastern language to be tackled. It’s only a matter of how fast it can learn the ropes.”

No moat. Not kiasuism, not Singlish, not the routine task you’re clinging to. Given time, the machine learns every rope.

That sounds like defeat until you turn it over. If nothing fixed is safe, then stop betting on AI’s limits - and start betting on your own capacity to keep making what hasn’t been made yet. Don’t protect the output. Protect the thing that generates new outputs. The wall always falls. The river keeps moving.

What This Looks Like for the Rest of Us

I’m not a comic artist. Maybe you’re not either. But the three moves translate cleanly, because they were never really about drawing.

• Stay fluid - be hard to reduce to a single repeatable function.

• Orchestrate - own the judgment, not just the task.

• Don’t argue with time - invest in the part of you that learns, not the skill that’s about to be commoditised.

None of that is use more AI, and none of it is hide from AI. It’s a third posture. Neither Mr Kiasu nor Mr FOMO. The person who has made peace with the weather and gone back to work.

There’s a coda that makes this literal. The Mr Kiasu robot - Singlish-speaking, built with Dex-Lab and the National Library - goes on more permanent display this year. You can go and ask the character himself what he thinks about AI. He didn’t fight the technology and he didn’t chase it. He absorbed it, and got a body out of the deal. Co-existence, drawn.

Which Character Are You?

Three parts, one man, and answers I didn’t expect to find through a comic artist.

We’re all somewhere in Johnny’s love triangle right now. Mr Kiasu. Mr FOMO. And Ai Swee, the object of their affections. Some of us are Mr Kiasu, bracing against loss. Some of us are Mr FOMO, sprinting so we don’t get left behind. Both are chasing. Both are afraid. Neither is living.

Johnny’s been the third character for thirty-five years - the one who didn’t panic and didn’t hype, who stayed fluid, kept the concept moving, and trusted the part of him that wasn’t a pattern to keep finding new forms.

So here’s the question I’m leaving with. Not will AI take my job, and not am I using enough of it. The better one:

What’s the part that you are still growing?

More soon. Different artist next time.

#Art #GenerativeAI #MrKiasu #FutureofArt #Singapore #CreativeIndustries #AIandArt

I knew nothing about this world.

I still remember my PhD supervisor’s reaction when I showed him yet another shiny concept from the latest paper. Unimpressed, he said: “Let’s start with the task.” I didn’t appreciate it then. Three years of fiddling with data types and tasks - networks, time series, multimodal data; regression, classification, forecasting - made me see the point.

That’s what this series is about. Something quite simple. Perhaps too simple. But I thought worth writing down.

How to see AI through the lens of data types and tasks. I call it “Thinking in AI”.

Nothing fancy.

But why does it matter then? In this age of LLMs and Agentic AI? Why not just jump ahead to prompt engineering, context engineering, or what one calls harnesses these days?

Because prompts are brittle. Context just means what goes into the Large Language Model. LLMs are not the only type of model in AI. And a framework for thinking is more durable than a harness.

AI has always been about classification, regression, generation, ranking, forecasting and many more tasks.

LLMs wrap over those tasks, in a more generalized manner (hence the name foundation models). Agents are another - an LLM orchestrating tools and other LLMs on tasks.

Strip the headline away and the data types and tasks are still the ones that were there ten years ago.

It all comes down to a few frameworks - for data, tasks, and methods - and how they compose into what we now call generative AI and agents.

So if you’re here for the frontier, you are in the wrong place. This series is deliberately about the boring fundamentals.

If you already know the fundamentals of the journey from machine and deep learning to LLMs and Agentic AI, then this might also be too simple for you.

For everyone else, I hope this is helpful in some way/

Start with the task

I call it “Prompt and Pray.”

You’ve probably sat through one of these. A workshop, a webinar, a demo. Someone shows a clever prompt. Everyone marvels at the output. Another prompt. Another marvel. A chuckle when the model gets something wrong. A tweak, and we’re back to marveling.

No one stops to ask what the model is actually doing. What data it’s seeing. What task it thinks it’s solving. Why it works on this example but might not on the next.

That frustrates me. Not because prompting isn’t useful. Because it skips every step that lets you understand the result.

So I wrote this. A short series on the part of AI that doesn’t expire when the next model drops.

Boring fundamentals, I promised. Here’s the most boring one. Before reaching for a model, write down what you’re actually trying to do.

Three questions.

1. What data do I have?

2. What task am I doing?

3. Which method fits?

Most AI problems start with those three. Not how to use the latest shiny new AI model.

Same problem, different roles

The use cases below look different. But are they really?

Different contexts. The same underlying job: turn a flood of messy inputs into one decision.

Take the portfolio analyst.

From data to tasks

Two tempting responses to all this.

1. Throw it all at an LLM and shrug when it hallucinates.

2. Build a bespoke solution from scratch.

The useful next question is “which tasks am I doing?” Once the tasks are clear, the method question gets much smaller. The method might not even be AI.

We can usually map use cases and problems to specific tasks. Some examples.

Each does a single, clear job. Sentiment reads tone. Extraction pulls out the tickers and names. Retrieval finds the past episodes that are similar. Forecasting projects the future.

A use case or solution to a problem usually chains some of these together.

Even LLMs and Agentic AI work this way. A general LLM seems to break the frame - one model, any task. But the task didn’t disappear; it moved into the prompt. Ask for a summary and it summarises. Ask for sentiment and it classifies. Change the prompt and the context, and you’ve changed the task. An agent just chains those prompts - planning, retrieval, reasoning, generation. Each step has its own data type, its own task, its own method. Decomposition is how agents work under the hood.

The same move works beyond the analyst. The journalist chasing a story, the clinician reaching a diagnosis, the lawyer finding the case that matters - each breaks down into the same handful of tasks. Different roles. Same approach.

Where this goes next

The rest of the series dives deeper.

Part I - Components. The math behind any AI model. The data types it sees. The tasks it does.

Part II - Setups. How those components compose into machine learning, deep learning, generative AI, and agentic AI. Different names, same parts.

Part III - Data Types. Five worked examples of the framework. Tabular, text, image, networks, time series.

• Tabular. Why a small machine learning model on a 40-year-old spreadsheet can still beat billion-parameter models at most real decisions.

• Text. How a five-step language pipeline collapsed into a single model.

• Images. How an invisible change to a handful of pixels turns a stop sign into a speed-limit sign.

• Networks. What AlphaFold and your org chart have in common.

Time series. Why your ECG and a stock chart are the same kind of problem.

When LLMs first became accessible and popular in 2022, one easy way to tell if someone was selling snakeoil was if they claimed that either ChatGPT or LLMs could forecast or predict something in the future. Someone who said that obviously did not understand how LLMs then worked, and were essentially hallucinating or bullshitting with great confidence.

LLMs are trained on text data. To forecast, you are working in the domain of time series data. Text and time series are both sequences, but they have fundamentally different characteristics.

Things have evolved since. Quite a lot actually. I would listen more patiently now for the details if someone said he or she used LLMs for forecasting due to shifts in the field. However, there is still a clear distinction between LLMs for language or text, compared to time series foundational models inspired by LLMs.

Let me explain this. In 4 short acts.

Act I: The Classical World of Time Series Modelling

For decades, way before neural networks were practically usable, forecasting or predictions with time series data was the domain of statisticians and econometricians. Time series data is fundamentally different from tabular, image or text data. You can shuffle rows in tables, mix up images, or rephrase text, and the meaning would still be largely intact.

The first, but not unique, property of time series is sequence. Mixing up the sequential order of time series data renders it meaningless. This is what it has in common with text.

This is the world of models like ARIMA. Understanding such models provides a clear understanding of what matters for time series data.

The AutoRegressive Integrated Moving Average (ARIMA) models and its variants dominated time series analysis for ages. They captured the essential insights needed to analyse or make predictions with time series data.

The core ideas:

• AutoRegressive (AR): Current values depend on previous values

• Integrated (I): Many series become predictable after differencing

• Moving Average (MA): Current values depend on errors of previous predictions

Conceptually, every time series could be understood as a combination of level, trend, seasonality, cycles and some noise. This is what makes it different from text.

Y = LEVEL + TREND + SEASONALITY + NOISE

Seasonal ARIMA (SARIMA) handled recurring seasonal patterns. Vector AutoRegression (VAR) tackled multiple related series. Holt-Winters exponential smoothing handled trends and seasons differently. For volatile financial series, Engle’s ARCH and Bollerslev’s GARCH gave us proper models for time-varying volatility - the technical term is conditional heteroskedasticity, but the everyday intuition is that the size of the typical move on a given day is itself a moving target. Harvey’s structural time series models decompose a series into latent components - level, trend, seasonal, cycle - each modelled as its own stochastic process, with the Kalman filter doing the inference behind the scenes. And modern decomposition extensions like MSTL (Multiple Seasonal-Trend decomposition using Loess) handle multiple seasonalities at once - think hourly demand with daily, weekly, and annual cycles all stacked together.

If you want to start playing with these classical models, R’s forecast package (auto.arima, ets, stl) is the easy on-ramp, and Python’s statsforecast from Nixtla gives you the same family of models. Meta’s Prophet popularised an additive-decomposition approach for forecasting and sits closer to the classical end of the spectrum despite being open-sourced by a deep-learning lab.

Another fundamental characteristic of time series that is critical is the degree of stationarity. Much more than text or images (words only change meaning slowly over time, same for images), time series in many domains continually evolve, and are what we call non-stationary.*

Recall ‘I’ for integration in ARIMA? That step leads to a differencing operation that allows the time series to be more stationary, and hence more predictable.

The classical world of time series forecasting, because of this focus on the underlying patterns of trends, seasons etc., was inherently explainable.

That’s Act I. But before I move on to Act II, I thought it would be useful to mention that forecasting is not the only task you can apply to time series. You can also nowcast (predict current unknown values with time series data to date, like GDP); classify time series patterns, detect outliers for time series, and so on and so forth. But the fundamental characteristics of time series data that need to be taken into account remain the same.

Act II: Machine and Deep Learning’s Struggles with Time

The power of machine learning models in the last decade meant many tried to use machine learning models such as support vector machines, random forests, and boosting tree models for time series forecasting. But these models did not fit naturally with time series data. Not that it could not work, but it was a hit and miss.

The natural question then was, why switch to these significantly more complex machine learning models when the classical models were good enough?

Then came advances in computer vision and natural language processing driven by deep learning. Computer scientists being computer scientists, they started looking for new domains to test these models on. Naturally, given the importance of time series data in many commercial and financial settings, computer scientists started using these models for different tasks on time series data.

But here’s the thing - you can’t just throw time series data at a regular deep learning model and expect magic. Remember the properties - sequence, patterns, stationarity?

Sequence models for natural language processing like Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks made a lot of sense. DA-RNN introduced dual-stage attention for input and temporal selection. Time-aware LSTMs handled irregular sampling via decay gates. Temporal Convolutional Networks showed that convolutional architectures could match or beat RNNs for many sequence tasks.

Answering the question of how to remember important information from way back in the sequence, while forgetting irrelevant noise applied equally to text and time series data.

There were many papers that focused on adapting RNNs, LSTMs, and even CNNs (designed for image data) to time series data. I would say the results were ambivalent. Sometimes you got fantastic results, sometimes a simple classical model would beat the complex deep learning model handily in a fraction of the time and computing power. A reviewer once asked me why a model I developed and trained for time series modelling was so large in size. It was only 12m parameters (miniscule compared to the billions and trillions for LLMs), but you get the idea. Probabilistic recurrent forecasters like DeepAR and deep state-space variants became quite popular. NeuralProphet - the DL successor to Meta’s Prophet, swapping the additive components for neural network blocks - sounded promising but empirically failed to beat plain ETS on standard benchmarks. The “DL doesn’t automatically win” pattern was already showing up.

So was it a pointless exercise? Not really.

One of the key differences between classical and deep learning time series models was that deep learning models could utilise multimodal time series data (time series of text, audio, sensor events, images, networks), whereas classical models were largely restricted to numerical time series.

There are also many other aspects of time series that we could focus on aside from sequence for time series, distinctly different from natural language or image data.

For example:

• Different time scales: Time series that change by the second, hourly, daily, quarterly and so on and so forth.

• Varying signal quality: Time series data that are inherently noisier than natural language or image data.

• Relationship dynamics: The importance of interactions between different time series.

There are many more. And deep learning models gave us a lot of flexibility to explore these characteristics.

In the early 2020s, even before ChatGPT, but after the seminal “Attention is All You Need” moment, many papers also explored the use of attention-based transformers for time series data.

It was a logical pairing. We use positional encodings to encode the position of tokens (or words) in transformers for text. Why couldn’t we do the same for time series steps?

A whole family came out of this - N-BEATS for interpretable basis expansion, the Temporal Fusion Transformer (TFT) for multi-horizon attention, Informer for long sequences, TS Transformer for self-supervised representation learning, Autoformer for decomposition + auto-correlation, and PatchTST. The PatchTST trick - that you can chop a series into patches and treat them like tokens - would turn out to be a conceptual seed for Act III. CoST pushed contrastive disentanglement of seasonal and trend representations.

In parallel, multimodal work fused text and prices for financial forecasting - event-driven stock prediction such as “Listening to Chaotic Whispers”, which added news attention. Fine-grained event typologies, stock embeddings from news + price history, hierarchical multi-task models for earnings-call volatility, REST’s relational event-driven framework, and FAST’s news-and-tweet time-aware network all extended this line.

I published several papers in this domain - using numerical and text time series with evolving networks. KECE combined knowledge graphs with numerical and textual time series. GLAM distinguished between global and local temporal patterns with adaptive curriculum learning to handle noise. GAME designed latent sequence encoders for multimodal data of different frequencies. DynMix used dynamic self-supervised learning with implicit and explicit network views, while DynScan learned slot concepts to handle non-stationary multimodal streams. These models showed strong results on financial forecasting, portfolio optimization, and ESG predictions, but they were highly specialized transformer models trained for specific tasks.

They were not general purpose or foundational models. Just using a transformer does not qualify. A general purpose or foundational model needs to be usable across different tasks.

But there were folks already researching these, even prior to ChatGPT coming onto the scene in 2022.

For readers who want to play with this era of models, the Python libraries darts (which also contains classical models), Nixtla’s neuralforecast, and HuggingFace’s transformer-based time-series pipelines are good entry points.

Act III: Foundation Models Learn the Language of Time

Interestingly, while I was researching multimodal time series models focused on networks, one of my PHD mates (Gerald Woo) was doing groundbreaking work on one of the first foundational models for time series data inspired by LLMs architectures.

This was the Moirai time series foundation model for universal forecasting. We attended each other’s research presentations once in a while, and I found his work fascinating. But I was already at the tail end of my PHD, so too late to switch tack.

Since then there have been many more foundational models for time series data inspired by LLMs architectures.

Now that you know all the special characteristics of time series, you would know that one cannot just throw a time series model into a transformer or an LLM and expect some magic to happen.

One key challenge was how to convert infinite numerical possibilities into a finite vocabulary that transformers can process. Language has a finite vocabulary, but not time series.

Different teams solved this differently:

Amazon’s Chronos quantizes continuous values into 4,096 discrete bins - essentially creating a “vocabulary” for time series. Google’s TimesFM treats time segments as “patches” like image processing. Salesforce’s MOIRAI uses multiple patch sizes for different temporal frequencies. There are other such models, but the fundamental issues being solved are similar. Address the tokenisation of time series, collate a large cross domain dataset, adjust the transformer architecture to address the unique characteristics of time series data.

In the months since the first version of this piece, the foundation-model wave has not slowed. But the field has clearly forked into two camps. Both are still very much “Act III” - they tokenize numerical time series and stick a transformer on top. They differ in where the parameters come from.

Camp A - Tokenize numbers, train a transformer from scratch.

This is the original recipe. The more recent successors mostly keep it and refine it:

• Chronos-2 - Amazon’s 2025 follow-up; adds in-context learning across related series and covariate-informed forecasting.

• ChronosX - extends Chronos to handle exogenous variables.

• MOIRAI-2 - simpler architecture, better data, generalises better than v1.

• MOIRAI-MoE - sparse mixture-of-experts variant; specialises across frequencies.

• Lag-Llama - open-source probabilistic TSFM.

• TimeGPT-1 - Nixtla’s offering.

Camp B - Don’t pretrain a TSFM. Reprogram a frozen LLM.

The intuition: LLMs already cost hundreds of millions to train. Maybe you don’t need to start from scratch - just teach the existing LLM to see numbers.

• Time-LLM - patches the series, reprograms each patch via cross-attention with text prototypes, freezes the LLM body, trains only the input projection and output head.

• LLMTime - encodes the series as a string of digits and lets the LLM autoregress. No training.

• GPT4TS / OFA - frozen GPT-2, fine-tune only the layer norms and positional embeddings.

• PatchInstruct - patch tokenisation + decomposition + neighbour augmentation as a prompting strategy.

What’s the point of foundational models for time series?

To me, the holy grail is probably few or zero-shot forecasting. Train once on massive time series datasets, then gain the ability to perform a range of tasks - forecast sales, detect equipment anomalies, or classify patterns across entirely new domains without additional training.

Still an unsolved problem I feel.

The key characteristics of time series data that we mentioned earlier are still key (such as sequence dependency, non-stationarity, varying frequencies, signal-to-noise ratios, and domain-specific patterns), and unlike text data, the nature of time series data in different domains (finance, healthcare, energy, retail, manufacturing, climate) can be vastly different and evolve significantly across time. We will talk a bit more about this below.

Act IV: The Agentic Turn - Forecasting as Reasoning

Acts I, II, and III all share an assumption. They assume forecasting is a single step problem. You feed in history, the model spits out the forecast or prediction.

The new wave has a few branches.

First, we can fuse different types of time series data at foundational model level across steps. From News to Forecast uses LLM agents to iteratively filter news, classify events by effect horizon, and fine-tune an LLM to emit digit sequences alongside a reflection loop.

Second, we can add reasoning. Training LLMs showed that reasoning at inference time - generating long chains of thought before producing an answer - improves performance on maths and code. Does it improve performance on time-series forecasting? TimeReasoner wraps series + timestamps + contextual features into a hybrid prompt, feeds it to a slow-thinking LLM, and explores three reasoning strategies - making the LLM think about the series before answering.

And finally, add multiple steps to the mix. AlphaCast - a training-free three-stage workflow (Investigator → Generator → Reflector) that mirrors how an expert forecaster works: prepare context, predict, critique, refine.

Conclusion

The only conclusion is that there is no conclusion yet. The jury is still out.

For a long time, a good place to look at the state of time series models were the Makridakis Competitions. M1-M3 (1980s-90s) were won by classical methods. M4 (2020) was won by Slawek Smyl’s hybrid ES-RNN - exponential smoothing married to LSTM - not by pure deep learning. M5 (2022) was won by LightGBM (gradient-boosted trees) with feature engineering, not deep architectures. M6 (2025) pushed into finance and reported that most teams underperformed simple benchmarks. The recurring lesson across forty years of competitions: the gains attributed to deep learning are usually gains from feature engineering, ensembling, or hybrid approaches. A example of a new leaderboard for foundation models is GIFT-Eval - 24 datasets over 144,000 time series and 177 million data points, spanning seven domains, 10 frequencies, multivariate inputs, and prediction lengths ranging from short to long-term forecasts.

But even when these foundation models top the leaderboards, the picture could be misleading.

Rethinking Evaluation in the Era of TSFMs found that benchmark scores are inflated - the test data often overlaps with the training data, either directly or because similar time periods appear in both. Strip the overlap out and the impressive numbers shrink.

Re(Visiting) TSFMs in Finance put the leading foundation models against decades of stock returns from markets around the world. The result: off the shelf, they did not beat ordinary baselines. Even fine-tuning didn’t close the gap. The only thing that worked was re-training the model from scratch on financial data - at which point you’ve essentially built a domain-specific model, not used a “foundation” one.

Multimodal and agentic forecasting are even harder to judge. Rethinking Multimodal TSF Evaluation points out that many “text helps forecasting” benchmarks are flawed - for example, the news used in testing has often already leaked into the model’s training data.

And that’s where we are now. Lots of progress, but still many open questions and uncertainties.

References

Act I - Classical time-series models, surveys, and competitions

• Forecasting: Principles and Practice - https://otexts.com/fpp3/ - Modern, accessible, free treatment of the whole classical lineage. Best starting point.

• Automatic Time Series Forecasting: The forecast Package for R - https://www.jstatsoft.org/v027/i03 - *Origin of auto.arima.

• Forecasting Seasonals and Trends by Exponentially Weighted Moving Averages - https://www.sciencedirect.com/science/article/abs/pii/S0169207003001134

• Exponential Smoothing: The State of the Art - Part II - https://www.bauer.uh.edu/egardner/3301H%20Operations%20Management/ESG%20Publications/2006%20Exp.%20Sm.%20State%20of%20the%20art%20-%20Part%20II.pdf.

• Macroeconomics and Reality - https://www.pauldeng.com/pdf/Sims%20macroeconomics%20and%20reality.pdf

• Autoregressive Conditional Heteroskedasticity with Estimates of the Variance of UK Inflation - https://www.jstor.org/stable/1912773

• The Story of GARCH: A Personal Odyssey - https://public.econ.duke.edu/~boller/Papers/GARCH_JoE_2023.pdf .

• Forecasting, Structural Time Series Models and the Kalman Filter - https://www.cambridge.org/core/books/forecasting-structural-time-series-models-and-the-kalman-filter/CE5E112570A56960601760E786A5E631

• MSTL: A Seasonal-Trend Decomposition Algorithm for Time Series with Multiple Seasonal Patterns - https://arxiv.org/abs/2107.13462

• StatsForecast / Nixtla - https://github.com/Nixtla/statsforecast

• The M4 Competition: 100,000 time series and 61 forecasting methods - https://www.sciencedirect.com/science/article/abs/pii/S0169207019301128

• The M5 Accuracy Competition: Results, Findings and Conclusions - https://www.sciencedirect.com/science/article/pii/S0169207021001874

• Forecasting with Gradient Boosted Trees: M5 Uncertainty Winner - https://www.sciencedirect.com/science/article/abs/pii/S0169207021002090

• The M6 Forecasting Competition: Bridging the Gap between Forecasting and Investment Decisions - https://arxiv.org/abs/2310.13357

Act II - RNN / LSTM / Transformer-era and multimodal cluster

• DA-RNN: Dual-Stage Attention-Based RNN - https://arxiv.org/abs/1704.02971

• Patient Subtyping via Time-Aware LSTM Networks (T-LSTM) - https://dl.acm.org/doi/10.1145/3097983.3097997

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (TCN) - https://arxiv.org/abs/1803.01271

• DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks - https://arxiv.org/abs/1704.04110

• Deep State Space Models for Time Series Forecasting - https://papers.nips.cc/paper/2018/hash/5cf68969fb67aa6082363a6d4e6468e2-Abstract.html

• NeuralProphet: Explainable Forecasting at Scale - https://arxiv.org/abs/2111.15397

• A Hybrid Method of Exponential Smoothing and Recurrent Neural Networks for Time Series Forecasting (ES-RNN, M4 winner) - https://www.sciencedirect.com/science/article/abs/pii/S0169207019301153

• N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Forecasting - https://arxiv.org/abs/1905.10437

• Temporal Fusion Transformers for Interpretable Multi-Horizon Time Series Forecasting (TFT) - https://arxiv.org/abs/1912.09363

• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting - https://arxiv.org/abs/2012.07436

• A Transformer-based Framework for Multivariate Time Series Representation Learning - https://arxiv.org/abs/2010.02803

• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (PatchTST) - https://arxiv.org/abs/2211.14730

• CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting - https://arxiv.org/abs/2202.01575

• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting - https://arxiv.org/abs/2106.13008

• Deep Learning for Event-Driven Stock Prediction - https://www.ijcai.org/Proceedings/15/Papers/329.pdf

• Listening to Chaotic Whispers: A Deep Learning Framework for News-Oriented Stock Trend Prediction - https://arxiv.org/abs/1712.02136

• Incorporating Fine-grained Events in Stock Movement Prediction - https://aclanthology.org/D19-5106/

• Stock Embeddings Acquired from News Articles and Price History, and an Application to Portfolio Optimization - https://aclanthology.org/2020.acl-main.307/

• HTML: Hierarchical Transformer-based Multi-task Learning for Volatility Prediction - https://dl.acm.org/doi/10.1145/3366423.3380128

• REST: Relational Event-driven Stock Trend Forecasting - https://arxiv.org/abs/2102.07372

• FAST: Financial News and Tweet Based Time Aware Network for Stock Trading - https://aclanthology.org/2021.eacl-main.185/

• Learning Knowledge-Enriched Company Embeddings for Investment Management - https://dl.acm.org/doi/abs/10.1145/3490354.3494390

• Investment and Risk Management with Online News and Heterogeneous Networks - https://dl.acm.org/doi/full/10.1145/3532858

• Guided Attention Multimodal Multitask Financial Forecasting - https://aclanthology.org/2022.acl-long.437/

• Dynamic Multimodal Implicit and Explicit Networks for Multiple Financial Tasks - https://ieeexplore.ieee.org/abstract/document/10020722/

• Dynamic Multimodal Slot Concepts from the Web - https://dl.acm.org/doi/full/10.1145/3663674

Act III - Foundation models (Camp A: train-from-scratch TSFMs)

• Unified Training of Universal Time Series Forecasting Transformers (MOIRAI) - https://arxiv.org/abs/2402.02592

• Chronos: Learning the Language of Time Series - https://arxiv.org/abs/2403.07815

• A Decoder-Only Foundation Model for Time-Series Forecasting (TimesFM) - https://research.google/blog/a-decoder-only-foundation-model-for-time-series-forecasting/

• Chronos-2: From Univariate to Universal Forecasting - https://arxiv.org/abs/2510.15821

• ChronosX: Extending Time-Series Foundation Models to Support Exogenous Variables -

AI Horizon Forecast

ChronosX: Extending Time-Series Foundation Models to Support Exogenous Variables

Foundation models excel in univariate time-series benchmarks…

Read more

a year ago · 6 likes · Nikos Kafritsas

• MOIRAI 2.0: When Less Is More for Time Series Forecasting - https://arxiv.org/abs/2511.11698

• MOIRAI-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts - https://arxiv.org/abs/2410.10469

• Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting - https://arxiv.org/abs/2310.08278

• TimeGPT-1 - https://arxiv.org/abs/2310.03589

Act III - Foundation models (Camp B: LLM-reprogramming)

• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models - https://arxiv.org/abs/2310.01728

• Large Language Models Are Zero-Shot Time Series Forecasters (LLMTime) - https://arxiv.org/abs/2310.07820

• One Fits All: Power General Time Series Analysis by Pretrained LM (GPT4TS / OFA) - https://arxiv.org/abs/2302.11939

• PatchInstruct: Forecasting Time Series with LLMs via Patch-Based Prompting and Decomposition - https://arxiv.org/abs/2506.12953

Act IV - Multimodal text + TS

• From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection - https://arxiv.org/abs/2409.17515

• Unlocking the Value of Text: Event-Driven Reasoning and Multi-Level Alignment for Time Series Forecasting - https://arxiv.org/abs/2603.15452

Act IV - Agentic & reasoning frameworks

• AlphaCast: A Human Wisdom-LLM Intelligence Co-Reasoning Framework for Interactive Time Series Forecasting - https://arxiv.org/abs/2511.08947

• Empowering Time Series Forecasting with LLM-Agents (DCATS) - https://arxiv.org/abs/2508.04231

• Can Slow-Thinking LLMs Reason Over Time? Empirical Studies in Time Series Forecasting (TimeReasoner) - https://arxiv.org/abs/2505.24511

Evaluation

• GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation - https://arxiv.org/abs/2410.10393 · leaderboard at https://tsfm.ai/benchmarks/gift-eval

• Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challenges - https://arxiv.org/abs/2510.13654

• Rethinking Multimodal Time-Series Forecasting Evaluation - https://openreview.net/forum?id=Z1TMV4bGuu

• Re(Visiting) Time Series Foundation Models in Finance - https://arxiv.org/abs/2511.18578