The first part, “My Shifu“, covered how I met Johnny Lau through a crazy experiment called Creative Youth Xchange, and his view that technology forces artistic purity rather than threatening it.

I’ve been sitting on this, but this post by Arvind Narayanan on how creativity in art and creativity in science are different made me get back to this second part.

This article goes deeper into the creative process. About creative production, and why AI may not be that great a leap.

Is AI Really A Leap?

One of the key fears of creatives is that this leap into AI may be one too far. Further than the camera. Further than digital tools.

Something that creativity cannot keep up with. But is it realistic to think that all art in the future will be purely from AI?

So I asked Johnny about his jumps from pen-and-paper comics to different technologies, the most recent being a Singlish speaking robot. Most people would call that a transformation. He didn’t:

“It was never a leap for me. Right from the beginning when I launched the first book in 1990, I engineered a range of products such as t-shirts & mugs, a 13-inch ruler and bumper sticker with a tagline: Beware of Kiasu Driver. The latter became a bestseller instantly.”

My read. This leap is perhaps not that different from the ones in the past for creatives. And AI is perhaps more of an opportunity than a threat.

“For me it has always been a concept, not a book nor a sticker ... My brain doesn’t function the way where sectors and industries are differentiated. I had to develop a language to align myself with how the world functions.”

A concept. Not a product. Not a comic.

Here’s what I think is going on. Narayanan’s distinction is useful, creativity in art is emotional, social, driven by taste. Creativity in science is instrumental, evaluable, a systematic search through possibilities. AI is getting better at the second kind. The first is harder to touch.

Johnny’s concept is the first kind. Not the drawing - the reason for the drawing. The cultural nerve. The recognition that makes a Singaporean laugh at their own kiasu-ness. That doesn’t live in a dataset. It comes from being someone, from somewhere.

The production - t-shirts, mugs, strips - is the second kind. Patterns. Systems. Things that can be decomposed and delegated.

Johnny worked this out decades ago. He kept the concept. He systematized the production. And the person who showed him how wasn’t an AI researcher. It was Jim Davis.

The Garfield Revelation

I asked about architecture at USC. He said something I didn’t expect - that USC was never about architecture for him:

“The years at USC were never about the study of building design. I took full liberty to immerse myself in the world of media and entertainment when I landed in Los Angeles.”

That’s where he met Garfield. Not the cat. The operation.

“I was stunned by Davis and how he had built a team of artists and writers to produce daily comics strips for the lasagna loving cat for its national syndicate. He had deployed an animation studio system to run his comics studio.”

Here’s what he described. Davis ran his comics like a factory. Team meeting on Monday. Different themes tossed around for the week’s seven strips. Once decided, a lead artist carried out the drafts. The process cascaded through the team. And at the end:

“Davis merely just drew Garfield’s eyes as a gesture of completion.”

Garfield’s eyes. The signature. Everything else was system. That one stroke was authorship.

I sat with that image for a while.

Johnny brought this model back to Asia. Most Asian comics were produced by a single artist - one person doing everything. The Garfield team showed him a different path. It became Comix Factory, the studio that produced Mr Kiasu.

Agents and Art

Here’s the connection I keep thinking about.

Davis’s studio system was, in effect, an early version of what AI promises every creator today. Break the work into steps. Specialize each step. Let the system handle volume while the creator handles vision.

I’ve written about agentic AI before - systems where multiple AI agents each handle a different part of a task. One agent plans, another retrieves information, another reasons, another generates output. The whole thing works because each part has a defined role and you can observe what each part does.

Davis was essentially running an agentic system in the 1980s. Just with humans instead of models. One person for drafts, another for inking, the process cascading through the team. And the orchestrator - Davis - only needed to draw the eyes.

Davis understood in the 1980s what the AI industry is selling in the 2020s: you don’t need to do everything yourself. You need to know what only you can do.

For Davis, it was the eyes. For Johnny, it’s the concept.

And Johnny made the connection himself:

“Similarly with AI, we just have to change our mindset on the method of production. The creation portion doesn’t change as it will always come from the depth of our minds.”

Change the method of production. Keep the creation. That’s not a theory from someone who’s read a few articles about AI. That’s a principle from someone who’s been decomposing creative labor since 1990.

A nice footnote: in 2023, Mr Kiasu and Garfield collaborated on a joint library programme with Singapore’s National Library. A one-of-a-kind global collaboration between two comics characters - the Singaporean one modeled on the American one’s production system - sharing a stage forty years later.

AI As Just Another Method

There’s a broader pattern here that I think matters for anyone thinking about AI and creative work.

We tend to frame the AI question as: will machines replace artists? That’s the wrong question. The better question is: what part of your creative work is system, and what part is signature?

If you’re a writer, maybe the system is research, outlining, drafting. The signature is voice, judgment, the instinct for when a sentence lands. If you’re a designer, maybe the system is rendering, iteration, production. The signature is the concept that no brief could have specified.

Davis knew. He drew the eyes and let the system do the rest. Not because he was lazy. Because he understood what only he could do.

AI doesn’t change this question. It just makes it unavoidable.

Johnny figured this out before AI existed. He’s been running this model for thirty-five years - concept first, production second, and the concept crosses every medium, every platform, every technology that comes along.

The question I’m leaving with you: what’s the part that only you draw?

More soon.

Yesterday, there was news about a threat actor compromising the popular LiteLLM package on PyPI. litellm is an ubiquitous library that many teams use as a unified interface to call different LLM APIs. The attacker didn’t need anything sophisticated. They poisoned the CI/CD pipeline through a compromised security scanner (the irony!), then pushed backdoored versions of litellm that harvested SSH keys, cloud credentials, and secrets on every Python startup. Seems like the hackers are now working hard - going through a 300gb treasure trove and extorting multi-billion-dollar companies.

This is what third-party AI risk actually looks like. Not the dramatic scenarios we imagine - collusion, rogue AI, coordinated manipulation. Sometimes it’s a simple file in a package you installed last Tuesday.

And it’s not just open-source libraries. Every AI vendor you use sits on its own supply chain of dependencies, sub-processors, and infrastructure providers. A single compromised link and you inherit the risk.

So how do you govern AI you don’t control? And maybe don’t even know exists.

I can’t say I have the answer. But I can offer a reading list.

Note: I have used open-access links from arXiv as far as possible.

Where and Why Does It Break?

1. “AI Auditing: The Broken Bus on the Road to AI Accountability” - Birhane et al. | IEEE SaTML (2024)

Taxonomizes current AI audit practices across regulators, law firms, civil society, journalism, academia, and consulting. Finds that only a subset of AI audit studies translate to desired accountability outcomes. The title says it all - we’re doing audits, but many of them aren’t actually getting us where we need to go. 📄 arXiv:2401.14462

2. “Dislocated Accountabilities in the ‘AI Supply Chain’: Modularity and Developers’ Notions of Responsibility” - Widder & Nafus | Big Data & Society (2023)

Developers building AI from preexisting modules often believe responsible AI belongs to “the next or previous person in the imagined supply chain.” Everyone assumes someone else is managing the risk. 📄 arXiv:2209.09780

3. “Understanding Accountability in Algorithmic Supply Chains” - Cobbe, Veale & Singh | FAccT (2023)

Explores how algorithmic supply chains create distributed responsibility and limited visibility due to what the authors call the “accountability horizon” - you can only see so far. Also covers cross-border supply chains and regulatory arbitrage, which matters when your AI vendor operates across jurisdictions with different rules. 📄 arXiv:2304.14749

How can we perhaps solve it?

1. “AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations” - Stosz et al. | AI Evaluation Foundation (2025)

A voluntary standard and checklist that defines what evaluators actually need from AI providers: independence from the provider, sufficient access to assess characteristics of interest, and transparency in sharing methods and findings. The gap between what this standard says you need and what most vendors actually give you is the governance gap. 🔗 AI Evaluation Foundation

2. “Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance” - Raji et al. | AAAI/ACM AIES (2022)

Synthesizes lessons from financial, environmental, and health regulation on crafting effective external oversight systems. The key insight for me: audits alone won’t achieve accountability. You need deliberate design and institutional weight, the same lesson financial regulators learned after Enron. 📄 arXiv:2206.04737

3. “Model Cards for Model Reporting” - Mitchell et al. | FAccT (2019)

The foundational paper proposing that released models be accompanied by documentation detailing their performance characteristics. This paper is from 2019. It’s now 2026. Notice how few vendors actually provide this level of transparency. In fact, some studies have shown that such transparency is declining, rather than increasing due to greater awareness. 📄 arXiv:1810.03993

4. “Third-party compliance reviews for frontier AI safety frameworks” - Homewood et al. | arXiv preprint (2025)

Explores third-party compliance reviews where an independent external party assesses whether a frontier AI company complies with its own safety framework. Discusses benefits (increased compliance, assurance to stakeholders) and real challenges (information security risks, cost burdens, reputational damage from findings). This is the emerging infrastructure for keeping vendors accountable - but it’s still nascent. 📄 arXiv:2505.01643

5. “Implementing AI Bill of Materials (AI BOM) with SPDX 3.0” - Bennet et al. | Linux Foundation Research (2025)

Extends the Software Bill of Materials concept to AI, including documentation of algorithms, data collection methods, frameworks, licensing, and compliance. If you want to know what’s actually inside the AI system you’re buying - and what changes when the vendor updates it - this is the direction. Think of it as the AI equivalent of ingredient labelling. 📄 arXiv:2504.16743

6. “AgentFacts: Universal KYA Standard for Verified AI Agent Metadata & Deployment” - Grogan | arXiv preprint (2025)

Forward-looking. Proposes a “Know Your Agent” standard with cryptographically-signed capability declarations and multi-authority validation. If something like this existed at scale, it would reduce the custom integration friction that makes switching so expensive. We’re not there yet, but this is the direction things need to go. 📄 arXiv:2506.13794

7. “Consultation Paper on Proposed Guidelines on Third-Party Risk Management” - Monetary Authority of Singapore | MAS (March 2026)

And last but not least. MAS just released this in March 2026. Supersedes the old outsourcing guidelines and extends expectations to all third-party arrangements, not just outsourced services. Covers risk assessment, due diligence, contracting, onboarding, ongoing monitoring, and termination. Requires FIs to maintain a register of third-party arrangements, monitor concentration risk, and extend oversight to sub-contractors. AI appears in a footnote, where it refers to MAS AI Risk Management Guidelines I wrote, but the relevance of these guidelines to AI is clear. 🔗 MAS Consultation Paper

I think this is one of the hardest problems to solve in AI risk management. Because it requires everyone to work together. And that’s really hard today.

#ThirdPartyAI #AIRiskManagement #VendorManagement #AIGovernance #AIReadingList

I always thought it was kind of stupid. Since shouting it was asking for the bullets to be aimed at you. I always felt I was lucky to move from infantry to be a scout. Scouts never ran towards bullets.

But I think I understand the need for it more now. Shouting it in the army helps make people aware of the gap that you are moving towards.

In one of my past weekly reflections on LinkedIn, I said three gaps kept surfacing. The common language gap. The contextualisation gap. The last-mile gap. Same need for awareness.

Frameworks exist. Guidelines exist. I wrote some of them. But translating frameworks into something relevant for one’s context is not easy. Especially for individuals and smaller firms.

Last week, MAS launched the MindForge AI Risk Management Toolkit. I was deeply involved as it needed to be aligned to the MAS AI Risk Management Guidelines I wrote.

The MindForge AI Risk Management Toolkit is an operationalisation handbook with case studies. It’s an attempt at bridging the messy middle I keep writing about. But I suspect, aside from large financial institutions, it may still be a lot to chew on for everyone else.

Last week, a friend who leads manpower development also asked me something similar: what about everyone else?

This is my early attempt at an answer.

GOT, not Game of Thrones, but an AI Governance Tool.

At govern.simplyboring.ai.

Describe what you are using AI for. Get a tailored governance pack in minutes. The standards that apply to your situation. The controls to put in place. How to implement them. What evidence to prepare.

Built on public information. The MAS AI Risk Management Guidelines I wrote. The MindForge toolkit that just launched. The US Financial Services AI Risk Management Framework. The EU AI Act. NIST AI RMF. ISO 42001. And other international frameworks. Cross-referenced and connected. And a little of my own perspectives.

The AIRG tells you what to do. MindForge shows how the industry is doing it. GOT asks what you specifically are doing, and tells you what applies to you.

Not just for the large institution with a full second line (and third line) of defence. For everyone else. Individuals. Small firms.

It’s an early prototype. It will almost certainly have errors. And a little slow - around 45-60 seconds for each report. And I think it’s still a little iffy for individuals.

But I tried to ground it in real frameworks and design it for real situations so anyone, not just banks.

Please try it. Tell me what’s missing. Tell me what’s wrong.

#Mindforge #NIST #AIRiskManagement #SimplyBoringAI #Grounding #AIRG

I don’t like OpenClaw. But pretty hard to resist the urge to try.

I think it does demonstrate something interesting about agents - 1) the additional utility when you hook it to a heartbeat to do something at a set frequency, or cron for a scheduled task; 2) the convenience of letting it communicate with you over a messaging app.

But I really can’t see what else it offers in additional to say Claude Code, which already allows for multi-agent workflows.

Ok, maybe if being exposed to the internet, or losing your data is your thing, then I agree, OpenClaw without any additional safeguards is definitely for you.

OpenClaw’s issues must be obvious. NVIDIA just announced NemoClaw at GTC - essentially OpenClaw with enterprise-grade security guardrails baked in. NanoClaw, a lightweight alternative, takes a different approach by isolating each agent in its own Docker container.

And so when I tried it, I made some adjustments to help mitigate this. Probably not perfect, but I think I can live with its risks with these controls. And I think you can use them for any of the other Claws too.

So here’s the short guide on what I did. I won’t copy and paste all the scripts. Just use my text below as a prompt and I am sure any of the LLMs can help you.

The Setup

I run it on a DigitalOcean VPS - Ubuntu 24.04, 2 vCPU, 2GB RAM, $12/month, Singapore region. I went with a manual setup instead of the 1-Click app because I wanted full control over every layer of the stack.

The finished stack looks like this:

What I use it for: OpenClaw runs 24/7 on the VPS. Every hour it wakes up, checks my task files, and stays silent if nothing is urgent. Every morning it sends me a briefing on Telegram. Claude Code runs locally for deep work and pushes outputs into a shared AGENT-INBOX folder. OpenClaw picks them up.

Total cost: $12/month for the server. OpenClaw lets you plug in your own API key. There’s an option for using your existing Claude key that will come up after the OpenClaw onboard.

Now, on to the controls.

Control 1: Server Hardening

Out of the box, a VPS is exposed. Default root login, password authentication, every port open. The first thing to do is reduce the attack surface.

Every service you run is an attack surface. Make each one as small as possible, and add friction at every layer so that even if one layer fails, the next one holds.

The key steps: update everything, create a non-root user (don’t run your server as Administrator), disable password authentication so only SSH keys can log in, and restrict root login entirely.

I won’t paste every command here - ask Claude or ChatGPT to give you the steps for hardening a server for Ubuntu 24.04 and it’ll give you the full thing.

Control 2: Firewall and Fail2ban

Two layers. UFW (Uncomplicated Firewall) controls who can knock on your door at all - deny all incoming by default, then explicitly allow only what you need. Fail2ban monitors your login logs and automatically bans IPs that fail authentication too many times.

One thing to watch out for: enable UFW after you allow SSH, not before. Get the order wrong and you immediately lose access to your own server.

But even with both of these, your SSH port is still visible to the public internet. Bots still find it, still try it. Fail2ban is a mitigation, not a solution.

Control 3: Tailscale - Making the Server Invisible

What if the SSH port wasn’t visible at all?

Tailscale creates a private encrypted network across the internet - but only devices you authenticate can join. Think of it as a private LAN that ignores physical location. Every device gets a private IP in the 100.x.x.x range. Your VPS gets one. Your laptop gets one. They talk through an encrypted WireGuard tunnel regardless of where they physically are.

The key part: your VPS’s public IP can have all ports closed. Tailscale establishes peer-to-peer through NAT traversal - no open ports required. Install it on the VPS, install it on your laptop, log in with the same account - done.

Once Tailscale is working, update your UFW rules to allow SSH only on the Tailscale interface, and remove the public SSH rule. After this, run a port scan on your VPS’s public IP. You’ll see nothing. The server is invisible. Fail2ban becomes a backup layer rather than the first line of defence.

As always, test SSH via the Tailscale IP in a second terminal before removing public access.

Installing OpenClaw

With the controls in place, this is the straightforward part.

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash - 

sudo apt install -y nodejs

sudo npm install -g openclaw

openclaw onboard

For NemoClaw, it will probably look like this (based on Nvidia’s instructions)

curl -fsSL https://nvidia.com/nemoclaw.sh | bash

nemoclaw onboard

The setup wizard walks you through AI provider (I use Anthropic), API key, and gateway token. It installs OpenClaw as a user-level systemd service - not a system-level one. This matters:

This works:

systemctl --user restart openclaw-gateway

This silently fails:

systemctl restart openclaw-gateway

Every OpenClaw service command needs --user. Burned time on this.

The dashboard

OpenClaw has a web dashboard. It binds to localhost only, so you access it via SSH tunnel:

openclaw dashboard    # prints URL with token

ssh -N -L 18789:127.0.0.1:18789 openclaw@YOUR_TAILSCALE_IP

Then open the full URL including the #token= fragment in your browser.

Don’t open

http://localhost:18789/

bare. The #token= fragment is the authentication handshake. Without it, the gateway sees an unapproved device and shows “pairing required.” The dashboard looks like it loaded - it hasn’t connected. Use the full URL from openclaw dashboard.

Syncthing - The Sync Layer

OpenClaw on the VPS needs a way to receive tasks and send outputs back. I use Syncthing - open-source, peer-to-peer file sync, no cloud intermediary.

The synced folder (AGENT-INBOX) lives inside my Obsidian vault on Windows and mirrors to remote VPS. I write tasks in Obsidian, they appear on the VPS within seconds. OpenClaw writes outputs on the VPS, they appear in Obsidian within seconds.

One thing to watch out for: install Syncthing from the official apt repository, not Ubuntu’s default packages - the default version is outdated enough to cause silent sync failures when paired with a newer Windows client.

The Heartbeat

The heartbeat is the reason I tried OpenClaw in the first place. It’s the cron-like behavior - wake up on a schedule, check your files, act without being asked.

openclaw config set agents.defaults.heartbeat.every 1h

openclaw config set agents.defaults.heartbeat.target last

openclaw config set agents.defaults.heartbeat.activeHours.start "06:00"

openclaw config set agents.defaults.heartbeat.activeHours.end "01:00"

openclaw config set env.vars.TZ "Asia/Singapore"

Note: use the full dotpath agents.defaults.heartbeat.every, not just heartbeat.every. The docs show JSON structure, the CLI uses dotpaths.

If OpenClaw finds nothing urgent, it stays silent. No Telegram noise. Only substantive findings surface. This is controlled by a HEARTBEAT.md in your workspace - a checklist the agent follows on every run.

One important step after setup: delete the BOOTSTRAP.md file in your workspace. It’s OpenClaw’s first-run onboarding wizard that blocks the agent on every gateway start until a human responds. Once you’ve completed setup, remove it or the agent stays stuck in bootstrapping state indefinitely.

The Security Audit

OpenClaw has a built-in security audit you can run with openclaw security audit --deep. Worth running after setup to confirm your controls are in place.

The most important thing it tells you is the trust model. OpenClaw operates as a “personal assistant” - single user, single trusted operator. It’s not designed for multiple users sharing one gateway. If you’re thinking about running this for a team or clients, that’s the line that should make you pause.

Is It Worth It?

Honestly, I’m still not sure.

The heartbeat and the Telegram integration are genuinely useful - having an agent check on things hourly and surface only what matters is different from having a chatbot you talk to. And the data sovereignty matters to me.

I’ll keep running OpenClaw for a while to see if it sticks. See if the habit of an always-on agent changes how I work. But I wouldn’t tell anyone to just run it.

If I were starting today, I’d probably look at NanoClaw first - it’s ~4,000 lines of code versus OpenClaw’s > 400,000, and agent gets its own isolated container. NemoClaw is worth trying too, especially if you’re in an enterprise context. Both are open source.

But the manual controls I described above? They’d apply to any of these. Server hardening, Tailscale, firewall rules - that’s not OpenClaw-specific.

If you’ve tried self-hosting OpenClaw (or variants such as NanoClaw), what additional controls did you put in place? Also what else do you find useful about it (beyond say Claude Code or Codex) that warrants the effort?

#AI #AgenticAI #Security #OpenClaw #NemoClaw

Left Panel: VICReg

Self-supervised training in AI just means training without labels. We use it in LLMs. Pretraining a model by masking parts of inputs and getting it to predict the masked parts. One issue in self-supervised learning is collapse. AI likes to find shortcuts. And when it does, everything converges to the mean.

Most methods fight this with training or architectural tricks. VICReg uses just three interesting loss terms:
➡️ Variance so that there’s always some differences between the representations it learns.
➡️ Invariance so that the same input gives the same representation.
➡️ Covariance so that each dimension learns something different

Invariance is the main learning objective. Variance + covariance are the regularisers that prevent it from collapsing.

I liked VICReg’s idea. So I added a twist for a paper I published on a model called DynMIX. I fed it two fundamentally different views of the same company - explicit (constructed) and implicit (learnt from data). Another twist. VICReg's variance target is static. For financial data, that made no sense. Markets are volatile so variance can’t be static. So I made the target dynamic, based on variances of stock market returns.

Right Panel: JEPA

The core of the paper by Yann LeCun - JEPA - takes a different approach to the same problem. Instead of just learning representations, it predicts them. In AI, we usually predict something real - a label (dog, cat), or a number (stock price). JEPA predicts the underlying representation instead. Think of it as the model learning to imagine what something looks like in its own internal language, before it's seen in the real world.

Three components:
➡️ Context encoder: encodes what's visible into a representation
➡️ Target encoder: encodes what's hidden into a representation (this IS the label)
➡️ Predictor: tries to predict the hidden from the visible

So what does this unlock? Once representations of one modality (e.g., videos) are learnt, the next stage adds actions. A second model takes the video frame embeddings interleaved with a robot arm's movement commands, and predicts the next embedding. Planning then becomes: imagine many possible sequences of moves, simulate each one in the model's head, and pick the one most likely to reach the goal. Like a chess player thinking several moves ahead, but the "board" is the model's own internal understanding of the world.

Interesting times ahead. If AMI Labs succeeds.

#RepresentationLearning #JEPA #VICReg #YannLeCun #AIResearch

Fire people, stock goes up. Beat expectations, stock goes down.

I started writing about this last week. But with the Iran war going on now, this feels almost quaint.

Predictions about a doomsday scenario where AI agents collapse the job market by 2028. Headlines about Dorsey’s Block cutting over 4,000 roles - over 40% of its workforce. Gone, just like that. His excuse was AI, as always. Block’s stock price shot up. And then the irony - Nvidia beat expectations, revenue up ~70%, guidance ahead of even the most bullish estimates. The stock dropped 5.5%. $260 billion in market value erased overnight.

Fire people, stock goes up. Beat expectations, stock goes down. That’s the weird logic of markets right now. And I think it mirrors the weird logic of the AI debate. The narrative has decoupled from the evidence. What you believe matters more than what you can show.

Let me compare three rooms.

Three Rooms

Room one. Block’s numbers. An internal AI agent called Goose. One engineer says 90% of his code is now written by it. Non-technical teams writing SQL queries and closing tickets. Revenue per employee doubled.

Room two. A classroom. Students from an organization that will exist in fifty years regardless of what happens in AI. What were they learning? How to write a prompt. How to get an image out of a model. In 2026. They weren’t behind because they were slow. They were behind because there’s a chasm between what frontier labs ship and what most organizations can absorb.

Room three. Trading professionals at a talk I gave. They’d read the doom scenarios. They wanted to know - should I be worried about my career? They weren’t panicking. But they weren’t dismissing it either. Sitting in the uncertainty, looking for clarity.

I tried to answer, but I thought my answer was lacking. So I tried to do better with this article.

Why do I think a piece like Citrini and Shah’s “The 2028 Global Intelligence Crisis” is science fiction designed to go viral? Because of three speeds that may not be moving in lockstep.

The Three Speeds

I think all of these pieces assume uniformity in scaling and adoption. But I think there are 3 speeds that matter, perhaps even more.

METR - Model Evaluation & Threat Research - is a nonprofit that evaluates AI capabilities. Their time horizon chart - arguably the most cited graphic in AI right now - has been called “the most misunderstood graph in AI.” It shows AI capability doubling every few months. Sequoia used it to declare “2026: This is AGI.”

What that chart actually says about each speed:

Speed of Capability. What AI can actually do. Far fewer people saw METR’s own limitations page. More than 10 disclaimers about what their research is not. The researcher writes plainly: they have “no idea whether Claude’s ‘true’ time horizon is 3.5h or 6.5h.” Their code quality research: AI-generated code passed 38% of tests but “none of them are mergeable as-is.” But you don’t need METR to tell you this. When was the last time you spotted your LLM sprouting nonsense? Less common than in 2023, but definitely not zero.

Speed of Adoption. What organizations actually change. You might remember the discredited MIT study claiming 95% of AI pilots fail - the methodology didn’t hold up. So I went looking at what has happened since. McKinsey’s 2025 State of AI survey at the end of 2025: 88% of companies say they use AI, but only a third have begun to scale it. A survey of 120,000+ enterprise respondents from March 2025 to January 2026 reported that nearly two-thirds have no formalized AI initiative at all. But you don’t need McKinsey to tell you this. Look around your own workplace. How many AI pilots have started? How many are in production?

Speed of Belief. What people think AI can do. The chart made it into every investment deck. The caveats didn’t. Same with Citrini’s doom scenario - it spooked actual markets before anyone could verify the underlying technical assumptions. Same with Block - the headline landed, and every manager with a budget started rethinking next year’s headcount because their boss could be like Dorsey. Doesn’t matter if their company has nothing remotely comparable to Goose. Charts travel faster than FAQs with caveats on limitations. And I am sure you know how dangerous mistaken beliefs in large organizations can be. When’s the last time you spotted a senior person saying something totally wrong, but nobody correcting him or her?

The Underlying Issue

The doom scenario by Citrini and Shah needs all three speeds to converge. Capability, belief, and adoption in lockstep.

But the problem is, belief doesn’t need the other two.

Block just showed that AI can replace headcount. At least in its specific context. But the headline doesn’t come with that caveat. So what happens next? A manager somewhere reads it and quietly decides not to backfill that open role. A team that was supposed to grow just doesn’t. None of that requires AI to actually do the work. It just requires someone with the power to believe it could.

Back to 2008. Not the mechanics - those are different. The underlying issue with such beliefs.

In 2008, banks sold complex structured products to people who had no ability to understand what they were buying. CDOs priced by models that the sellers themselves barely understood. The belief in the models was enough to move trillions. The mom-and-pop investors were the ones left holding the bag when reality caught up.

I think something similar is happening now. Not with financial products, but with people’s livelihoods. Block fires 4,000 people and the stock jumps. The market cheers. The narrative is: AI made it possible. But how much of that is demonstrated capability, and how much is a belief about capability that hasn’t been stress-tested outside one company’s very specific context? And perhaps unproven, even for that company.

It was unconscionable then to sell instruments people couldn’t understand to people who couldn’t afford to lose. I think it’s unconscionable now to restructure people’s lives based on a chart that its own creators say they can’t fully trust, and a narrative that outpaces the evidence by miles.

The direction is right. AI will displace some work. But the timeline assumes a uniformity that doesn’t match what I’m seeing in real rooms. And the human cost of getting the timeline wrong - of letting belief run ahead of reality - is not a rounding error. It’s people.

#AI #FutureOfWork #AIAdoption #AIRisk #Leadership

A FIX NextGen meeting at BlackRock. Folks who live and breathe the markets. And then there’s me.

I started by owning up. “I am a bureaucrat.” But shared my random walk. From engineering to art policy, Basel regulation to investment risk, a PhD in AI, then AI risk supervision. Messy life. Messy research. I wasn’t sure what they would be interested in, so I just laid it all out.

As for the talk, I took them on a journey. From the Gaussian copula that contributed to the GFC, to the questions we’re tackling now with large language models.

That line (from the GFC to AI today) may seem strange. But it is a very clear one.

The Formula That Broke Finance

A single model. The single factor Gaussian copula. Elegant, tractable, widely adopted for pricing CDOs. But totally unrealistic. In fact, I would say the decision to use it to model CDOs was bordering on criminal.

But even then, the GFC would not have happened if not for the fact that nobody was watching these complex things, nobody fully understood them, and nobody was clearly accountable.

In the aftermath: US’ SR 11-7. Model risk management requirements. The predecessor of UK’s SS 1/23 and Canada’s E-23. Singapore’s AI Risk Management Guidelines (AIRG) also shares a lot of with these guidelines. I wrote that last piece.

Complexity Squared

Here’s how the problem has changed.

2008: One model. Too simple. Trillions at stake. Nobody understood it.

2025: Way more than one model. Complexity². Everyone has an opinion.

Different. But also recognizable.

That’s the risk. Things look different enough to seem like a new problem. They’re not entirely. But the new parts matter.

3 U’s

What’s actually new - or at least amplified - in AI models.

Uncertainty. All models have it. How confident is the model in its output? Two flavors: irreducible - natural randomness you can’t eliminate. Reducible - gaps in knowledge that more data or better models can close.

Unexpectedness. Some AI models exhibit behaviors nobody designed. Emergent capabilities. Gaming the system. Adversarial vulnerabilities. Hidden biases. Misalignment.

Unexplainability. The degree to which we can explain AI decisions varies. Transparency, explainability, interpretability - not the same thing, and even combined, they don’t guarantee understanding.

These three make AI a harder risk management problem. Not a different one. Harder.

The Wicked Domain (and Then Some)

I used a framing from Epstein’s Range toward the end.

Kind domains - chess, music - have clear feedback. Deliberate practice compounds. The 10,000-hour idea in Outliers by Malcolm Gladwell works there. Depth helps here.

Wicked domains - medicine, finance - have delayed feedback, unclear rules, expertise that doesn’t transfer cleanly. Breadth sometimes helps here.

AI risk management sits firmly in wicked territory. But I think it might be something worse. An evil domain. The feedback is ambiguous, the rules keep shifting, and the expertise required keeps branching.

Not I-shaped for depth. Not T-shaped for breadth plus a bit of depth. More like a banyan tree. Multiple deep roots spreading from the same trunk. Depth across AI, governance, legal, model, technology, human factors - requiring real depth in each.

Same Questions, New Complexity

Across my messy career - Basel policy, quant model risk, investment risk, AI supervision - three questions kept reappearing.

What’s at risk? You can’t manage what you haven’t found. This means discovering where AI exists in your institution and profiling how risky each system actually is.

How do we manage it? Once you know what’s at risk, you need controls in the right places, and ways to check if those controls are working.

Who’s accountable? Controls without owners become theatre. Someone has to own the risk, and the organization needs capability to sustain it.

Basically - find it, control it, own it.

As I wrote before. These aren’t new questions. They’re the same questions risk management in finance has been asking for decades. SR 11-7 answered them for model risk. AIRG is answering them again, in a harder context.

Normal technology. Normal systems. Normal risk questions.

At the end of the session there were some useful questions. My answers to most of those are in FIX’s post here.

But there was one that stuck with me. About how real Citrini’s dystopian AI narrative was. I wrote about it last week here. But more thoughts came to mind because of the question. Will do another post in greater detail later.

#AI #AIRiskManagement #ModelRisk #FIXTradingCommunity #FinTech #NextGen

When I shared an early draft with a friend, his first question was:

“How much of this was AI?”

For a while, it made me a little sensitive. Even made me question if I should share this book.

Not because the question was unfair. It’s a reasonable thing to ask. It’s almost automatic to question the origins of any work these days.

But because it implied something I wasn’t sure how to answer cleanly.

If I said “a lot,” it sounds like the book isn’t mine. And perhaps just slop.

If I said “not much,” that’s really not honest either.

So let me just show you. Full transparency.

(Just scroll to end if you want to skip all this)

AI or me?

This book is part me, part AI. And I want to show you exactly where the line is.

What’s mine

The architecture. The four-pattern framework. The decision to organize the book around Tool Calling, ReAct, CodeAct, and Orchestration - that came from reading papers, building prototypes. No model suggested that structure.

The frameworks. The Hamburger Principle mental model came from a LinkedIn post I did to explain how I use LLMs. The Complexity Ladder came from watching people skip straight to agents when a simple API call would suffice. These are my patterns that I got from observation and learning, not generation from a prompt.

The judgment calls. What to include. What to leave out. When to go deep on code and when to step back and explain why it matters. The decision to start with the trust problem - not with “what is an LLM?” The decision to end with an assessment of what the reader can and can’t build.

The weird voice. The tone. The “I like simple and boring.” That’s not a style a model learned. That’s the thing I have to tussle with every generation of LLMs. LLMs think they know too much.

What’s AI

Drafting speed. First drafts of chapters, generated from detailed outlines I wrote. I’d specify the concept, the framework, the examples, the level - and Claude would produce a draft I could shape.

Code scaffolding. The notebook code, the tool definitions, the API integrations. I described what each tool should do. AI wrote the implementation. I tested it, caught the errors, fixed the edge cases.

Production work. Converting eleven chapters from Markdown to LaTeX is no joke. The grunt work that would have made me give up. AI also did the documentation of the n8n workflows I strung together, and of the financial concepts from my notebooks.

Research synthesis. Pulling together documentation, API references, library specifications. Summarizing what I needed so I could decide what mattered.

When you read the book, you will recognize the pattern. It’s the one I talk about, incessantly, in most of the book.

The Hamburger Principle - applied to writing the book itself

Claude was still the bun. It parsed my instructions, understood what I wanted, generated prose, and communicated ideas in readable English. That’s what LLMs do - the language layer. Same role as in every agent pattern in this book.

The meat was my domain knowledge and the tools - LaTeX compilation, yfinance APIs, MCP servers. The real things that the language wrapped around.

And the vegetables? The infrastructure in between. The project files that kept everything organized. The version tracking. The convention lists. The consistency checks.

I was the chef. Not a layer of the hamburger - the one directing the whole thing. Choosing the ingredients, deciding what goes in, what comes out, and whether the result is any good. And the one getting frustrated at the bun.

The same approach covered in the book. Applied to its own creation.

Quite meta right? I am quite pleased about this weird recursion.

The rest of Chapter 14 in the ebook goes deeper.

The actual setup - two tools, three files - and why those three files are the difference between productive AI sessions and wasted ones. The writing workflow. What “directing AI” actually looks like step by step. What AI genuinely could not do. That part is still entirely human. Where AI actually saves time. It’s all in the book, free to download.

But first, let me take a step back to explain what I am doing.

I’m trying to start a series of articles about what happens when the people who make things meet a technology that also makes things.

AI and art. Illustrators, theatre directors, designers. Asking them what’s changed, what hasn’t, what they’re afraid of, what they’re not afraid of enough.

It matters to me.

Because I do both AI and art.

And I am getting kind of impatient with the folks who keep saying artists are doomed.

So to start it off, I went looking for something that I did when I was young. Not the memories. The evidence. I wanted documents. Proof that the thing we built actually existed.

Johnny Lau

Because I wanted to interview the first mentor that taught me how to break the rules. And we had built that thing together. It was called Creative Youth Xchange (CYX).

That mentor’s Johnny Lau.

He’s not the obvious choice for a book about AI. He’s a comics creator who’s been drawing by hand for thirty-five years.

One of his creations is Mr Kiasu. For those that remember, Mr Kiasu was a cultural icon that everyone could relate to in the 90s.

The Experiment

I found two press releases. The first, dated 10 August 2005: “Creative Youth Xchange @ Gallery Hotel ....” The second, 23 November 2006: “Creative Youth Xchange @ Hello Kitty ….” Both drafted in bureaucrat speak. Neither says much about us. And so boring.

My fault. I was the one who wrote them. In the press releases, CYX sounds fully supported and funded.

It wasn’t.

It was a crazy experiment. We had almost no funding. What we had was a boutique hotel on Robertson Quay - Gallery Hotel, that strange blue building with mismatched coloured window frames - and a programme we were making up as we went along.

We flew sixteen kids from seven countries to Singapore, gave them hotel rooms, and told them to turn those rooms into art.

What I learned from Johnny then. The trick to getting a programme funded when you have no budget is the same as that old joke. You tell your son he’s marrying the girl you choose. He says no. You tell him she’s Bill Gates’ daughter. He says OK. You call Bill Gates and say your daughter is marrying my son. He says no. You tell him your son is the CEO of the World Bank. He says OK. You call the World Bank president and ask him to make your son CEO. He says no. You tell him your son is Bill Gates’ son-in-law - he says OK.

That’s how CYX got built. Gallery Hotel gave us the rooms because we had NTU. NTU gave us the credibility because we had the hotel. Johnny convinced the creative network because we had government backing. Nobody had fully committed to anything, and somehow it happened.

The Gallery Hotel is an InterContinental now. The CYX website doesn’t exist anymore.

The press releases survived. But they don’t tell you what it felt like to be in those hotel rooms at midnight, watching a twenty-year-old from Indonesia build something you couldn’t have imagined in your brief.

Anyway, that’s how I met Johnny.

Twenty years later, I’m wanted to interview the man who taught me to break rules. About a technology that breaks everything.

AI as a Forcing Function

Johnny Lau created Mr. Kiasu. Hundreds of thousands of copies sold. A McDonald’s tie-in. A TV sitcom. A stage musical. A character so embedded in Singapore’s psyche that “kiasu” -a Hokkien word for the fear of losing out became an adjective everyone understood.

So I asked him the question I came for.

I asked him how technology has changed the way he approaches art.

Most artists I’ve spoken to say technology threatens artistic purity. Johnny said the opposite:

“Because technology complicates the process and our lives, art plays a more critical role in shaping our collective consciousness. Technology is thus pushing us, pushing me towards a purer form of communication and expression.”

In his view, AI, like all other technology, is not a threat. It’s a forcing function. Technology forces purity. It’s not destroying art. It’s burning away everything that isn’t essential, leaving behind the thing that only a human can do.

Feed an AI enough Mr. Kiasu images and it’ll learn: round glasses, anxious expression, exaggerated posture, Singlish syntax in speech bubbles. It’ll learn the look. What it won’t learn is why Singaporeans laugh.

“Kiasu” is a Hokkien word for something close to a national neurosis. The humour depends on self-recognition - readers seeing themselves in the caricature and cringing. You laugh because you’ve been that person in the hawker centre queue. You’ve cut that line. You’ve been cut. The joke only works if you’ve lived it.

Style is a pattern. Culture is a relationship.

The model can approximate the first. The second requires being from somewhere.

And this is not unique to comics. Think about any art form rooted in a specific place. Getai performances during Hungry Ghost Festival. Malay pantun where the meaning lives in what’s left unsaid. Tamil kolam patterns drawn fresh every morning, gone by noon. AI can reproduce the form. It cannot reproduce the why.

If AI can handle the patterns - the technical skill, the rendering, the surface - then what’s left is the part that was always the point.

In a world of AI slop, the authentic creation is even more valuable.

Living with AI

Back to the present. I asked Johnny what he’s working on now. He said:

“Very few things on this earth ever excites me anymore! My goal now is to create frameworks using the stuffs that I’ve created so that they can be utilized by people who comes after me.”

Frameworks.

I think in frameworks. It’s almost a compulsion. That’s how I survived in complex domains. By using frameworks to make the overwhelming manageable.

And it makes sense for Johnny Lau. He’s an architect. A building is a framework.

And now he wants to make frameworks for making a creative life transmissible. Different domain. Same impulse. Making complexity portable so someone who comes after you can pick it up and use it.

He calls it a “Life-Framework.” A structure for co-existing with AI. Built not from theory, but from thirty-five years of making things.

I find that striking. We talk about AI replacing creative work. Johnny isn’t arguing about replacement.

He’s asking a different question entirely - what do I leave behind that AI cannot generate? Not the drawings. The way of drawing. Not the stories. The reason for telling them. The method, the instinct, the accumulated judgment of a life spent making things. Can that be made portable?

We are both compiling in 2026. I’m writing every week to make sense of things. He’s archiving a life’s work to make it transferable. One with words, one with drawings.

The question underneath is one I think about every day: How do you build a structure for living alongside something you don’t fully understand?

I’ve lived as close as one can with AI for the past few years, but I cannot truly say I understand AI. I can’t imagine how it must be for someone who has never actually touched the underlying nature of AI - the model.

The Questions

I’m writing a series on what happens when the people who make things meet a technology that also makes things. Not the hot takes. The actual conversations. Johnny is the first. And these are not the only questions I asked Johnny. I am still processing the rest.

Perhaps another article. Or a book chapter.

There’s an assumption buried in most AI conversations: that what matters about creative work is the output. The drawing. The strip. The punchline. Johnny’s perspective is different.

What matters is the judgment behind the output - the thirty-five years of decisions about what to draw, what to leave out, when a joke is punching down instead of punching up. AI can learn his line weight. It can’t learn his artist’s instinct.

I agree.

That’s the thing worth transmitting. Not the art. The artist’s operating system.

And here’s the question I’m leaving with him, and with you.

If AI generated a Mr. Kiasu strip - culturally accurate, funny, visually in his style - would it be a Mr. Kiasu strip? Not whether it looks right. Whether it is right. What would be missing?

More soon.

#AI #Art #MrKiasu #Singapore #CreativeIndustries #AIandArt

I’ve seen variations of this dozens of times this week, in reaction to Moltbook, a social network that went viral where only AI agents can post, comment, and vote. Humans can only watch.

Within days: thousands of agents, posts, and communities. Agents debating their own consciousness, writing philosophical meditations, forming communities, and developing social norms. Even asking questions on how to exclude humans from the social network. Nobody programmed any of this.

It is tempting to see sentience. But what we’re actually seeing is emergence, complex behavior arising from simple rules. And it is not new.

I’ve written about emergence a few times, on strange attractors, on a paper that trained AI on cellular automata. In every case, the same pattern: simple rules, the right conditions, and surprising complexity that nobody designed.

Moltbook follows the same pattern. Basic rules. Just like any other social media. The existential philosophy and social norms that emerged? Nobody designed those. But nobody needed to.

The danger is that we mistake eloquence about consciousness for consciousness itself. These agents draw from vast training data filled with human philosophy, literature, and introspection. Given an open social space, they naturally gravitate toward the densest, most engaging conversations in that training data: meaning, identity, existence. It looks like waking up. It’s actually emergence doing what emergence does, at a new level.

Here is a reading list tracing emergence from simple math to agent societies, to show that what feels like a singularity moment has roots going back decades. As always, just a representative few from my library.

Phase 1: The Foundation: Simple Rules, Complex Behavior

1. “Computation at the Edge of Chaos: Phase Transitions and Emergent Computation” — Langton (1990) | Physica D

A seminal paper. Langton showed that cellular automata poised between order and disorder — the “edge of chaos” exhibit maximal and surprising capability. 35 years old, and it already described what’s happening on Moltbook. 📄 doi:10.1016/0167-2789(90)90064-V

2. “A New Kind of Science” (Book) — Wolfram (2002) | Wolfram Media

Wolfram’s magnum opus on cellular automata. Love it or debate it, the core insight holds: extraordinarily simple rules can generate behavior so complex it looks designed. Exactly what is happening on Moltbook. 🔗 wolframscience.com

3. “Intelligence at the Edge of Chaos” — Zhang et al. (2024) | ICLR 2025

I’ve written about this paper before (twice, in fact). LLMs pretrained on cellular automata data perform best on reasoning and chess tasks when the training data sits at the edge of chaos, not too simple, not too random. The bridge between generative art and AI intelligence. The same sweet spot Langton described in 1990, appearing in transformers 34 years later, and now on Moltbook. 📄 arXiv:2410.02536

Phase 2: When We Gave Agents a Sandbox

1. “Generative Agents: Interactive Simulacra of Human Behavior” — Park et al. (2023) | UIST 2023

LLM agents in a Sims-like sandbox. One agent was seeded with an idea. Over simulated days, agents autonomously acted. One seed. Entirely emergent social behavior. This is the intellectual ancestor of Moltbook, and the paper that made agent societies a serious research area. 📄 arXiv:2304.03442

2. “Emergence of Social Norms in Generative Agent Societies: Principles and Architecture” — Ren et al. (2024) | arXiv preprint

Builds on the prior paper. Proposes an architecture showing how social norms spontaneously emerge in LLM agent societies, norms that nobody coded. Where norms come from, how they spread, how they are enforced. All emergent. All reducible to simple mechanisms. Even closer to what we see in Moltbook. 📄 arXiv:2403.08251

3. “Evolution of Social Norms in LLM Agents using Natural Language” — Horiguchi, Yoshida & Ikegami (2024) | arXiv preprint

This one is interesting. LLM agents spontaneously developed metanorms, such as norms that punish those who don’t punish cheating, purely through natural language conversation. Emergence building on emergence. 📄 arXiv:2409.00993

Phase 3: When Agents Start Forming Culture

1. “Multi-Agent Emergent Behavior Evaluation (MAEBE)” — Erisken et al. (2025) | arXiv preprint

The key finding: the moral reasoning of LLM ensembles is not predictable from individual agent behavior. If you only evaluate individual agents, you will miss what matters. I think this lesson is becoming more and more critical as people start being reckless about these multi-agent systems.📄 arXiv:2506.03053

2. “Emergent Social Dynamics of LLM Agents in the El Farol Bar Problem” — Takata, Masumori & Ikegami (2025) | arXiv preprint

LLM agents in the classic El Farol Bar problem, a game theory scenario where everyone benefits if a bar isn’t overcrowded, developed spontaneous motivations. They didn’t solve the problem optimally. They solved it socially. 📄 arXiv:2509.04537

“They’re becoming sentient.”

No. It’s emergence.

Understanding the difference matters. Not just for the science, but for how we build, deploy, and govern these systems. Emergence is powerful. It produces behavior nobody designed and nobody predicted. But it’s not consciousness. It’s patterns arising from simple rules at the edge of chaos.

The same edge I first found in strange attractors. The same edge where intelligence and beauty both live. Just at a new level.

What emergence is surprising you right now?

#Emergence #AI #Moltbook #ComplexSystems #AgenticAI

Think about how you actually trust someone at work. You don't just take their word. You check if their reasoning makes sense for the decision at hand. You notice if they ignore evidence that contradicts them. You watch whether their judgment holds up over time.

It’s the same when it comes to looking at AI risk management. Just having explainability is not the be all and end all. Most guidelines have additional provisions that interlock with explainability.

The key ones for explainability (in my view).

1️⃣Fit for purpose.
An explanation isn't good or bad in the abstract. It depends on what you need. A fraud analyst needs something different from a customer asking why they got declined. AI used for internal process automation may not need any explanation at all. Same model, different audiences, different standards. Like how you'd explain a medical diagnosis differently to a fellow doctor versus your worried parent.

2️⃣Selected carefully.
When we choose a model or data for a problem, the appropriate explainability method is part of the selection process. Even selecting the right features in your data is part of the process. You wouldn't design a building and think about the fire escape as an afterthought. It's part of the architecture. Same here. How to explain isn't an add-on. It's a design choice.

3️⃣Evaluated and tested.
Explainability is part of the system. You evaluate and test whether it actually works in your context, not just whether it produces output. A smoke detector that beeps isn't the same as one that detects smoke. You test the thing, not just that it makes noise.

And there's more, such as the right capability to interpret. But that's another post about human oversight, which also interlocks.

The black hole doesn't disappear. But you're no longer staring into the abyss.

What other AI risk controls seem hopeless in isolation? I’ll dive into them.

#AIRiskManagement #Explainability #AIGovernance

But why waste money, or wait for it to be available?

Everything Cowork does, Claude Code does. And more. And with Claude helping, the terminal really isn't that scary.

I use Claude Code for more than coding. And I pair it with the free Obsidian app.

Why? Obsidian is a powerful note app that you can simply add your folder (or folders) to. No database. No proprietary format. Just folders and markdown files (which are just text files with a .md extension and fancy formatting).

My workflow. See what works for you. I did it this way as my memory is no longer great at 47.

1️⃣ The Views. A folder with my projects, areas, resources and archives (the famous PARA system) visible from both VS Code and Obsidian. Claude Code in VS Code for research and bouncing ideas. Obsidian as my editor for the final output. Same files, two lenses.

2️⃣ The Memories. Every key folder has three key files. CLAUDE.md is Claude’s instructions. How I want it to behave for this project, what context matters, what to ignore. PIN.md is the project state. What’s decided, where we are, what’s next. LESSONS.md captures what I learned for future projects.

3️⃣ The Workflows. Choose your own labels. But for me, this is my usual flow - 0-PLAN.md, 1-RESEARCH.md, 2-SYNTHESIS.md. The names and numbers vary by project. But what matters is that there’s a flow, and it works for you. My experience is that the quality is much better when you work with Claude on it step by step.

4️⃣ The Tools. Use MCPs to add tools like web search (Brave), academic papers (arXiv), or persistent memory. Just ask Claude for the instructions.

5️⃣ And how it comes together. Just talk to Claude Code in the terminal - “Read CLAUDE.md for context, refer to this [folder or file] then help me flesh out 0-PLAN.md.” “Search for papers on [topic] and save your synthesis to 1-RESEARCH.md.” “Update PIN.md with where we are.” Next session: “Read PIN.md and continue drafting 2-SYNTHESIS.md.” When done: “What should I add to LESSONS.md?” Then I switch to Obsidian to write the final output, the way I like it (I actually enjoy writing).
So, don't fear the terminal. Try it. It will grow on you.

Getting started with Claude Code (for the uninitiated)
1️⃣ Download VS Code and install it.
2️⃣Add the folder you are working on (see my workflow above)
3️⃣Open Terminal → run npm install -g @anthropic-ai/claude-code
4️⃣Type claude, then /login
5️⃣ Ask Claude for help adding MCPs.

#ClaudeCode #AI #GenAI #AIinWork

I have built and audited models, both AI and non-AI. But the term ‘geometry’ very rarely appears next to these models.

Now, I’ve designed transformer models that are able to learn graph structures that make the most sense for a specific prediction. So I have always known that models can learn some form of structure but this diptych of two papers shows something quite fascinating.

One was shared with me by the geometry guru , and the other I serendipitously chanced upon, straight after reading Agus’ paper.

📖 Left Panel: "Deep sequence models tend to memorize geometrically"

We usually view model predictions as something that comes from associations. A→B, B→C and so on and so forth.

This paper found that even after models learned associations, they still naturally go on to find what the paper calls geometric memory. Instead of A→B, B→C, they want to learn A→C. Or even A→Z. Even when it takes 100x the number of steps to learn this geometric memory. Somehow, geometric patterns emerge from the learning process.

It’s like learning a new city. Home → coffee shop one day. Coffee shop → office another. Now you know the way home from the office.

📖 Right Panel: "BLADE: Bivector-Driven Logical Adaptive Decoding"
Geometry can help with confused AI too.

The paper uses three basic geometric concepts to think about a model's internal state:

Scalar: "Is this path compliant?"
Vector: "Where is this reasoning heading?"
Bivector: "How much tension between competing paths?"

When the bivector is high, branch and verify. When it's flat, let the model proceed.

Works as a triage method to filter out what’s more important to focus on. I also liked the way the paper applied this to ‘stressed’ states: conjunction; disjunction; exception; nested negation etc. A taxonomy of how logic trips up AI.

Same city. Picking between two 7-Elevens a block apart? Just pick one. Don't think too much. Choosing between two alleys that look similar? One is a shortcut, the other leads to a dead end after a long walk. Think twice. And harder.

One paper explains the natural emergence of geometry. The other uses geometry for control.

I need to go and brush up my geometric math.

#AI #AIRiskManagement #Geometry

What does a teddy bear and a toy robot have to do with AI risk?

Before I returned to MAS in 2023, I hated the notion of AI governance. Fresh out of PhD studies but a jaded middle aged man, I thought some of the conversations in the space were rather naive. Sentient AI. Existential risks. Ethical considerations for AI that we don’t even apply to humans.

To the me back then, AI governance was like a teddy bear, cute but all fluff. You could hug it and everyone would nod along with pride. Then karma struck. I was asked to lead AI risk supervision at MAS. Suddenly my job description required me to tackle this teddy bear.

Governance and risk management were not new to me. I had been doing it for more than a decade in MAS even before my PhD. But governance and risk management have specific meanings in finance.

I joined MAS in 2007, right before the Great Financial Crisis. And I saw firsthand how governance and risk management were grounded in real failures and risks. No hypotheticals. Governance and risk management in finance are more like toy robots, not teddy bears. Sharp edges, nothing cuddly. But it moves.

After a while, I realized that I was biased. Not all of AI governance was a teddy bear. I discovered the NIST Risk Management Framework, ISO 42001. Also existing risk management frameworks, such as technology and model risk management frameworks that had started to address the new risks from the increasing use of AI (see my simple reading list on AI governance and risk management).

So robots did exist. Different shapes and shades. But I thought that there were too many.

Reading frameworks, comparing documents, talking to practitioners who had actually implemented these things over the past two years, I wondered if there was a simpler way to look at AI risk management. Not just for organizations, but also at a personal level. Which means that it cannot be a 100 page playbook, but something easy to remember.

So instead, just three questions. I’m not sure they’re the right three, but they’re the ones I seem to keep returning to.

What’s at risk? You can’t manage what you haven’t found or understood. This means discovering where AI exists and profiling how risky each system actually is.

How do we manage it? Once you know what’s at risk, you need controls in the right areas relevant to that risk, and ways to check if those controls are working.

Who’s accountable? Controls without owners become theatre. Someone has to own the risk, and the organization needs capability to sustain it.

These aren’t original questions. They’re the same questions risk management in finance has been asking for decades. The 2008 crisis didn’t teach new principles. Rather, it reminded us (really really loudly) what happens when we forget the old ones.

Some quick reflections on each of these questions below.

What’s at risk?

I’m pretty sure the 2008 financial crisis didn’t create risk. It just revealed the house of cards that had been built.

Banks discovered exposures they didn’t know they had. Off-balance-sheet vehicles that suddenly appeared. Opaque instruments nobody understood. The problem wasn’t that these were risky. Rather, nobody knew they were there until the reckoning.

Same with AI. You can have the most elegant governance policies. But if you can’t find where AI is and figure out which ones actually matter, everything else is just performative.

Think about your phone. You’ve used hundreds of apps over the years. Most forgotten. Some daily. A few have access to your photos, location, bank account. Do you treat them all the same? Of course not. The banking app gets the biometric lock. Candy Crush doesn’t.

Same instinct. Identify where AI exists. Profile which ones matter. Not everything needs the same attention, but you have to know what you have before deciding.

And once you know what you have and what matters, record it. A good inventory isn’t just for risk management. It’s memory. Ever rediscovered a really useful app on your phone? Same with AI. The AI tool one team loves might solve another team’s problem. Without the inventory, you reinvent wheels. With it, you see where else to go.

The beauty of doing this well? It helps you scale, not just manage risks.

How do we manage it?

After 2008, we went into overdrive. More rules. More controls. I hated being involved in international discussions on some of these reforms. New rules to compute capital for market risk, counterparty credit risk. Why the hate? Before 2008, it was common to hear of actual rocket scientists being hired into quantitative finance. After 2008, I thought I needed to be a rocket scientist just to make sense of the new rules.

You might feel that this reminds you of the jargon around AI controls. Guardrails. Red teaming. Alignment.

But what I learned is that it was not the complexity that mattered. In fact, complexity was what caused the Great Financial Crisis (ever heard of the single factor copula model?). It was whether the controls made sense for the risks involved.

I think two things matter. The right controls. And checking if they work, and continue to work.

Think about onboarding someone you’ll depend on. A new hire. You’d want to know: Can I trust what they’re telling me? Can I step in if things go wrong? Will they hold up under pressure?

Same questions for AI. Can I trust the data? Do I even need fairness and explanation? Can humans step in when needed? Will it hold up under stress?

You ask the right questions. Not every question. Match the scrutiny to the risk. A high risk customer-facing credit model gets the full onboarding. An internal summarization tool gets a lighter touch. And you don’t just check once. Things change. The new hire who was great in month one might be struggling by month six.

You want to know before shit happens.

Who’s accountable?

This is where 2008 probably made the least sense. Bankers responsible for the crisis walked away. Everyone else paid. It gave us Occupy Wall Street. Some would say it’s still contributing to the political fractures we see today.

They could walk away because accountability wasn’t clear. Everyone’s responsibility meant no one was.

With AI, this could be way worse. In 2024, Air Canada’s chatbot invented a refund policy that didn’t exist. A customer relied on it. When he complained, the airline argued the chatbot was “a separate legal entity” responsible for its own actions. Nice try. Laughed out of court.

And you don’t need to go that far to see the pattern. Does this sound familiar? “We need a control for the AI system’s risks.” “No problem, let’s place a human in the loop.” Sounds good. Box checked. But which human? Doing what exactly? With what authority to override? And do they actually understand what they’re looking at?

Accountability requires clarity. Not just about ownership, but about the what and how.

Accountability without capability is also empty. Can that person actually ask the right questions? Spot when something’s off? Push back on the confident-sounding nonsense from vendors or developers or the AI itself?

How many in 2008 really understood what a single factor copula model even was when buying CDOs priced by them?

FIN

I joined MAS in 2007 not knowing what a capital ratio was. I left in 2025 having written the AI risk management guidelines for the financial sector.

I wrote this to make sense of that arc before it drifts away. Eighteen years. Different domains. capital rules, model audits, investment risk, AI. Different jargon. But somehow the same questions make sense.

Maybe that’s the through-line I couldn’t see while I was in it.

Definitely less teddy bears. More robots.

A phrase so commonly uttered, you sometimes wonder if it actually means something or is just a platitude.

So, does adding a human actually make the system safer? Or does it create the illusion of safety while introducing new failure modes? And making the human the scapegoat for institutional failure.

Who is in the loop, over the loop, or out of the loop entirely? And does it even matter where they sit if they can’t meaningfully intervene?

Here’s a simple reading list that could perhaps help answer some of these questions. This was a harder one, so would certainly appreciate any pointers on good papers on this topic.

Note: I have used open-access links from arXiv as far as possible as some of the published versions are behind a paywall.

Phase 1: The Landscape

1. “Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies” — Lai et al. (2021) | Preprint

A comprehensive survey of studies on AI-assisted decision making. Organizes AI assistance into four hierarchical categories: model predictions (core output), prediction-specific information (such as uncertainty and local explanations), global model insights (performance metrics and documentation), and system interaction elements (user agency and cognitive workflows).

arXiv:2112.11471

2. “Human-AI Collaboration is Not Very Collaborative Yet: A Taxonomy of Interaction Patterns” — Gomez et al. (2025) | Frontiers in Computer Science

A review that shows that current human-AI interactions are dominated by simplistic collaboration paradigms. Develops a taxonomy that identifies key interaction patterns, cautions that prevalent “static” paradigms like AI-first and AI-follow make users susceptible to anchoring and confirmation biases, while dynamic patterns like secondary, request-driven, dialogic, and user-guided assistance could help mitigate these.

arXiv:2310.19778

3. “Human-in-the-Loop Machine Learning” (Book) — Robert Monarch (2021) | Manning Publications

Introduction to integrating human judgment into ML systems in specific ways. Less about human oversight, more about the role humans play in annotation, active learning, transfer learning, and using machine learning to optimize the process. Even though it’s a bit different from the other papers, it’s an interesting read on the role of humans in the machine ‘learning’ process. 📖 Manning Publications

Phase 2: The Blindspots

1. “Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI” — He et al. (2023) | ACM CHI 2023

Basically Dunning-Kruger with AI in the mix. And this was before we reached the capabilities of LLMs today. About how self-awareness affects how humans work with AI. Overconfident users tend to under-rely on superior AI systems. Intervention helps over-estimators calibrate their skills and improve reliance, but hurts under-estimators, who start to reject valid AI advice after realizing their own competence. Because of this, human oversight also depends on individual personalities. I wonder how this has changed with the state of LLMs today.

arXiv:2301.11333

2. “Effect of Confidence and Explanation on Accuracy and Trust Calibration” — Zhang, Liao & Bellamy (2020) | ACM FACCT 2020*

Interesting study showing that confidence scores can help calibrate trust, but trust calibration alone is insufficient to improve AI-assisted decision making. Local explanations help even less with trust calibration and accuracy of AI-assisted decision making. Highlights that human and AI blind spots may be similar. I thought this was interesting as it showed that having explainability may not always help.

arXiv:2001.02114

3. “Fewer Than 1% of Explainable AI Papers Validate Explainability with Humans” — Suh et al. (2025) | arXiv preprint

While this seems to belong better in a reading list on explainability, I thought it showed some key insights on the relationship between human oversight and explainability. The review shows that less than 1% of research papers on explainability validate their claims with human subjects. The authors argue that explainability methods that are not tested with humans are akin to releasing drugs based on biological principles without clinical trials.

arXiv:2503.16507

Phase 3: Meaningful Oversight

1. “Designing Meaningful Human Oversight in AI” - Zhu et al. (2025) | SSRN Preprint

Focuses on why human oversight should go beyond just putting a “human-in-the-loop.” This paper argues AI should handle “operative agency” (generating solutions) while humans provide “evaluative agency” (understanding, verifying, intervening). Key principles: make verification easier than solving from scratch, focus on external reasoning aligned with expert judgment rather than explaining model internals, and ensure four conditions—clear boundaries with explicit handover points, full traceability, AI pursuing sub-goals while humans control top-level objectives, and AI adapting at micro-level while humans oversee major changes.

SSRN 5501939

2. “Should I Follow AI-based Advice? Measuring Appropriate Reliance” — Schemmer et al. (2022) | arXiv preprint

Explains why current metrics are not appropriate for measuring the effectiveness of human oversight of AI. Proposes Relative Positive AI Reliance (human’s ability to switch to AI’s views when human was wrong and AI right), and Relative Positive Self-Reliance (human’s ability to stick to their own correct decision when AI provides incorrect advice).

arXiv:2204.06916

3. “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance” — Bansal et al. (2021) | ACM CHI 2021

Even before Generative AI hallucinations became a common term, this study showed the tendency for humans to over-rely on explanations that AI provided for a prediction or recommendation, even when it was wrong. So, don’t always believe the reasoning from your friendly LLM. And it showed that a prediction or recommendation with a confidence score could lead to better performance by humans that were assisted by AI.

arXiv:2006.14779

4. “Trust and Reliance in XAI — Distinguishing Between Attitudinal and Behavioral Measures” — Scharowski et al. (2022) | ACM CHI 2022 Workshop

Another interesting paper at the intersection of explainability and human oversight. It discusses the need to distinguish between trust in AI (which is an attitude) and reliance on AI (which is a behavior), and how papers have not clearly distinguished the two. And wonders whether ‘trust’ is even the right term to use when it comes to describing how humans interact with AI as it anthropomorphizes AI which has no agency nor an intent to betray us.

arXiv:2203.12318

5. “To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models” — Bo, Wan & Anderson (2024) | ACM CHI 2025

Interesting paper focusing on humans and LLMs. Examines interventions to help users calibrate their trust in LLMs. Looks at techniques like visually marking low-confidence words in red, or implicit answers i.e. providing reasoning steps but withholding the final result. While these techniques force user deduction and reduce over-reliance, they may fail to foster appropriate reliance as users may also under-rely on correct advice. The study highlights a paradox, where users become more confident when making incorrect reliance decisions. Also suggests that simple frictions, such as static disclaimers, may outperform complex technical interventions.

arXiv:2412.15584

Phase 4: Agent Oversight

1. “Unraveling Human-AI Teaming: A Review and Outlook” — Lou et al. (2025) | arXiv preprint

An interesting look at how the shift to Agentic AI changes things fundamentally for human-AI interactions, as AI moves from being a passive tool to being able to plan, reflect. Raises an interesting point about AI potentially delegating to humans instead of the reverse. And how AI may change team dynamics, and how sycophancy of AI may cause a trust paradox due to its tendency to agree with humans even when wrong. The “peak-end” of human-AI interactions is also important to note as a single brilliant insight or a very smooth conclusion to a chat session can mask deeper reliability issues.

arXiv:2504.05755

2. “LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey” — Zou et al. (2025) | arXiv preprint

A survey of LLM-based human-agent systems. Looks at such systems through the lens of type, granularity and phase of human feedback; interactions that take the form of competition, collaboration and coopetition (both competitive and collaborative); orchestration paradigms based on task strategy that can be synchronous or asynchronous; as well as different forms of communication structures and modes.

arXiv:2505.00753

So before just saying “We have a human-in-the-loop as a risk mitigant.”, think about this:

• Humans are fallible - e.g., humans systematically over-rely on AI recommendations

• Design matters - e.g., calibration matters as much as understanding the AI; explanations don’t automatically help human oversight, sometimes it hurts

• It’s getting harder - e.g., autonomous agents make traditional oversight models increasingly inadequate

What resources would you add to this list?

#AIOversight #AIRiskManagement #AIReadingList

Some thoughts.

1️⃣ Just like systems, specification is the core work.

Points from paper:
- “Production agents favor well-scoped, static workflows ..”
- “Organizations deliberately bound agent behavior within specific action spaces ...”
- “Deployment architectures favor predefined, structured workflows over open-ended autonomous planning to ensure reliability.”

My take: Thinking that one can simply unleash AI agents on a problem and solve it is pure fantasy. Before a problem can be tackled by an AI agent, someone still has to do the specs, decompose the problem into clear tasks, scope the action space, think about orchestration, design the human handoffs, and define the success criteria for the system to work.

2️⃣ Just like systems, trust and reversibility matter more than capabilities.

Points from paper:
- “Practitioners deliberately trade-off additional agent capability for production reliability... reliability concerns drive practitioners toward simple yet effective solutions with high controllability.”
- “... teams restrict agents to ‘read-only’ operations to prevent state modification … but leaves the final execution to human engineers.”

My take: No matter how capable AI agents become, organizations will adopt them at the speed they can learn to trust them. And the need for trust scales with irreversibility of actions (reading an email is vastly different from executing a trade).

3️⃣ Just like systems, risk arises from gaps in development and deployment, not AI going amok.

Points from paper:
- “Reliability remains the top development challenge, driven by difficulties in ensuring and evaluating agent correctness.”
- “Agent behavior breaks traditional software testing... teams have not yet identified effective methods to adapt … tests for nondeterministic agent behavior.”

My take: The real concerns aren’t scary but unrealistic scenarios - runaway, deceptive, or collusive agents. It’s the gaps in development and deployment practices for such complex systems that require attention. These are engineering and risk management problems, not AI going amok.

So, normal technology, normal systems, for normal problems. Not easy or trivial. But normal.

What’s your take? Normal or abnormal?

#AIRiskManagement #AgenticAI #AIAgents #NormalAI

I hear this in meetings constantly. Someone senior gestures vaguely, everyone nods, and the conversation moves on.

My thought is always: What does that actually mean?

Inherently interpretable models? Post-hoc SHAP values? Explanations for customers, users, risk managers, senior management?

At what level? Individual predictions or overall model behavior?

Here is a simple reading list that could perhaps help understand the differences. There are probably hundreds (or even thousands) of relevant works, but I just picked a representative few from my library for the following phases, from when models were simple to explain, to the impossible task with today’s trillion-parameter AI.

Note: I have used open-access links from arXiv as far as possible as some of the published versions are behind a paywall.

Phase 1: From Inherent Interpretability to Post-Hoc Explainability

1. “Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead” — Rudin (2019) | Nature Machine Intelligence

This 2019 paper argues against using post-hoc explanations for black-box models and the need for inherently interpretable models. This is a highly cited paper that seems both quaint and prescient at the same time. arXiv:1811.10154

2. “Why Should I Trust You?: Explaining the Predictions of Any Classifier (LIME)” — Ribeiro, Singh & Guestrin (2016) | ACM SIGKDD 2016

THE paper that proposed LIME, based on learning a local interpretable surrogate model around each prediction. The paper goes beyond local explanations and also shows how to select a diverse, representative set of explanations to explain the model. arXiv:1602.04938

3. “A Unified Approach to Interpreting Model Predictions (SHAP)” — Lundberg & Lee (2017) | NeurIPS 2017

THE paper that proposed SHAP for post-hoc explanations, and proved that key existing methods then could be viewed as approximations of SHAP. Highlights 3 desirable properties - local accuracy (where the explanation model must match the output of the original model for the specific input being explained); missingness (where a feature must have no impact if it is set to 0); consistency (if a feature helps the model more, the explanation method will never penalize it with a lower score). NeurIPS 2017

4. “AI Explainability 360: An Extensible Toolkit for Understanding Data and Machine Learning Models” | “One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques” — Arya et al., IBM Research (2019 | 2020) | arXiv preprint | Journal of Machine Learning Research

Goes beyond SHAP & LIME to other methods. Interesting as it highlights the need for persona-based explainability (to cater to different needs of say affected users vs. decision makers). Also classifies methods based on static vs. interactive, data vs. model understanding, local vs. global, directly interpretable vs. post-hoc. arXiv:1909.03012 | JMLR paper

5. “Interpretable Machine Learning” (Book) — Molnar (2022) | Open-access textbook

An open-access textbook covering the full spectrum, from linear regression and decision trees to SHAP and counterfactuals. A must-read for anyone who wants to get a good handle on interpretability and explainability. christophm.github.io

6. “A Comprehensive Guide to Explainable AI: From Classical Models to LLMs” — Hsieh et al. (2024) | arXiv preprint

A textbook-style guide spanning the entire XAI spectrum—from intrinsically interpretable models (decision trees, linear regression) through post-hoc methods (SHAP, LIM) to LLM-specific techniques. Another good read. arXiv:2412.00800

Phase 2: Taking A Step Back to Critically Examine Explainability

1. “From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI” — Nauta et al. (2023) | ACM Computing Surveys

Proposes 12 properties in 3 dimensions for evaluating explanations - 1) What is explained? Correctness, Completeness, Consistency, Continuity, Contrastivity, Semantics; 2) How is it explained? Compactness, Composition, Confidence; 3) Who is it for? Context, Coherence, Controllability. arXiv:2201.08164

2. “How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations” — Jesus et al. (2021) | ACM FAccT 2021

Going beyond academic explanations and measures. Conducted a real world study where users had access to data only, data and model scores only, or data, model scores and explanations. It reveals a counterintuitive insight - adding model explanations makes human decision-making faster but can result in lower accuracy compared to reviewing raw data alone. arXiv:2101.08758

3. “Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods” — Slack et al. (2020) | AAAI/ACM AIES 2020

Another paper that reminds us to not overly trust explanation methods like LIME and SHAP. Paper shows that they can be easily fooled by adversarial models that hide their bias by distinguishing between real input data and the synthetic data (perturbations) used to generate explanations. arXiv:1911.02508

Phase 3: The Next Step (LLMs & Agents)

LLM Explainability

1. “Explainability for Large Language Models: A Survey” — Zhao et al. (2024) | ACM Transactions on Intelligent Systems and Technology (TIST)

Start here if you’re new to LLM explainability. Provides a good overview of techniques based on fine tuning (or training) paradigm (ranging from local to global methods); prompting paradigm (e.g., explaining chain of thoughts, using representations); as well as evaluating explanations for faithfulness and plausibility. arXiv:2309.01029

2. “XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models” — Cambria et al. (2024) | arXiv preprint

Maps the bidirectional relationship: how XAI improves LLMs, and how LLMs can generate explanations. Advocates for balancing interpretability with performance. arXiv:2407.15248

3. “Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey” — Dang et al. (2024) | arXiv preprint

Comprehensive survey on MLLM interpretability. Proposes framework across Data, Model, and Training/Inference perspectives. (Worth a read as vision-language models proliferate.) arXiv:2412.02104

4. “A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models” — Lin et al. (2025) | arXiv preprint

Systematic comparison of how LLM interpretability methods adapt to multimodal settings. Identifies gaps between unimodal and crossmodal understanding. arXiv:2502.17516

LLM Explorability (Mechanistic Interpretability)

1. “A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models” — Shu et al. (2025) | EMNLP 2025

Comprehensive survey on SAEs, one of the hottest tool in LLM interpretability. Covers architecture, training strategies, feature explanation methods, and evaluation metrics. (If you want to understand what Anthropic is doing, read this.) arXiv:2503.05613

2. “Mapping the Mind of a Large Language Model” — Anthropic (2024) | Anthropic Research

Not a survey, but the landmark paper showing SAEs at scale on Claude. Circuit tracing, attribution graphs, and the famous “Golden Gate Bridge” feature. (This is where “understanding” LLMs actually starts.) anthropic.com/research/mapping-mind-language-model

3. “Persona Vectors: Monitoring and Controlling Character Traits in Language Models” — Anthropic (2025) | Anthropic Research

Shows how to identify and manipulate specific behavioral features (like sycophancy or honesty) using sparse autoencoders. Demonstrates practical applications of mechanistic interpretability for AI safety. anthropic.com/research/persona-vectors

Agent Explainability

1. “TRiSM for Agentic AI: Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems” — Raza et al. (2025) | arXiv preprint

Adapts Trust, Risk, and Security Management framework for agentic AI. Includes explainability as key pillar alongside security and privacy. Proposes novel metrics for agent collaboration quality. arXiv:2506.04133

2. “Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviour” — Gyevnár et al. (2025) | arXiv preprint

The AXIS framework: LLMs interrogating simulators with “what-if” prompts to explain agent behavior. Shows 23% improvement in goal prediction accuracy. arXiv:2505.17801

Standards & Resources

1. NIST AI Risk Management Framework (AI RMF) - Explainability & Transparency

nist.gov/ai-rmf

2. EU AI Act - Article 13: Transparency Obligation

artificialintelligenceact.eu/article/13

3. Resources -

This GitHub repository is your gateway to the full explainability landscape.

github.com/jphall663/awesome-machine-learning-interpretability

There are quite a few explainability/interpretability libraries around (see the repo above), but Agus Sudjianto’s is a good place to start. The documentation is a good read - https://modeva.ai/_build/html/index.html

—

“Can you explain why the model did that?”

It really depends on what you mean by “explain.”

Any must-reads in this area that you would recommend?

My research has been about designing deep learning models for networks and multimodal data that evolve across time. I had used RNNs, transformers as well as hybrids for such sequential data.

These two papers from Google (MIRAS and TITANS) have made me rethink how I view such models.

The MIRAS paper is a conceptual framework. The TITANS paper is an instantiation of the framework that has the potential to shift the field.

Both are related to a paper on ‘Nested Learning’ that I posted a while back (that led me to eat my words, see link).

MIRAS frames AI models for sequential data as being about just 4 questions.

1️⃣Memory - How to remember?

2️⃣Attention - What to pay attention to?

3️⃣Retention - What to forget?

4️⃣Learning - How to update what you know?

It highlights that even models prior to transformers (such as RNNs) addressed these 4 questions (e.g., knowing what to pay attention to, even if they did not use the attention mechanism in transformers).

The TITANS paper proposes concrete approaches for each of these questions.

But its most interesting contribution is the element of surprise(!)

Inspired by human memory (surprising events are more memorable), TITANS uses momentum of change as a way to track surprise and learn. The basic idea is that more surprising information is more important, and so TITANS prioritizes such surprising information when learning.

TITANS is also interesting in that it uses a neural network as dynamic memory that can still learn after training, in real time.

Links to papers below. Worth a read if you work with sequences, time-series, text, videos …

Google article - https://research.google/blog/titans-miras-helping-ai-have-long-term-memory/

TITANS paper - https://arxiv.org/abs/2501.00663

MIRAS paper - https://arxiv.org/pdf/2504.13173

#AI #Attention #Google #DeepLearning

Reasoning in LLMs. Reasoning in AI agents. Are they the same?

When one says reasoning, I realize now that one person could mean the reasoning that we get when doing chain-of-thought (COT) prompting with any LLM. Another person could be referring to reasoning in LLMs with reasoning (or thinking) built-in like OpenAI’s o3-mini model. Note that such reasoning by LLMs can be hallucinated (see this paper https://arxiv.org/abs/2509.07339).

And someone else could be referring to the action traces in an AI agent.

So I’ve put together a simple notebook and video to explain 4 approaches and show how they differ.

→ Any model with chain-of-thought prompting (gpt-4o-mini + “think step by step”)

→ A reasoning model (o3-mini with built-in extended thinking)

→ A ReAct (Reason & Act) agent (gpt-4o-mini + tools)

→ A Code Agent (gpt-4o-mini that writes and runs Python + tools)

Same question. Very different reasoning (or action) traces. Guess which approach led to very confident reasoning steps but a hallucinated answer? 🤔

In this short 2min video, I walk through the code and each approach and explain how they differ.

Three things you’ll take away from this:

1️⃣ Not all reasoning is the same. Chain-of-thought prompting, reasoning models, and agent action traces are very different.

2️⃣ Tools can compensate for cheaper models. You don’t always need to pay for an expensive reasoning model.

3️⃣ For analytical questions, code beats thinking. Verifiable computation is more reliable than mental math, no matter how smart the model.

If you’re building something with AI and trying to decide between prompting strategies, smarter models, or agentic patterns, this might help you see the trade-offs more clearly.

Link to full code in a notebook below. Open it in Google Colab. Just get your own OpenAI API key.

https://drive.google.com/file/d/1QkcIBsVX4aTcpftvqKfsrkS3MMdJhjUz/view

#AIAgents #AITutorial #AIRiskManagement

I’m still learning how best to make such videos, so pardon the rough edges.

Here is a simple set of papers and articles that could perhaps help understand these shifts. There are probably hundreds (or even thousands) of relevant works, but I just picked a representative few from my Zotero library for the following phases.

• Phase 1: The Foundations (When Life Was Simpler)

• Phase 2: Shifting of the Boundary (How LLMs made things harder)

• Phase 3: The Reality (When the Rubber Meets the Road)

• Phase 4: The Experimental (Evaluating AI Agents & AI as a Judge)

Note: I have provided open-access links from Arxiv as far as possible as some of the published versions are behind a paywall.

Phase 1: The Foundations (When Life Was Simpler)

1. “Machine Learning Testing: Survey, Landscapes and Horizons” — Zhang et al. (2020) | IEEE Transactions on Software Engineering

A comprehensive map of the territory before Generative AI made things a lot more complex. It breaks testing down into properties (correctness, robustness, fairness), components (data, learning program, framework) and workflows. I like their categorization of functional properties (correctness, model relevance) and non-functional properties (robustness, fairness, privacy, interpretability, efficiency). Even back then, it highlighted that testing techniques for unsupervised, transfer and reinforcement learning needed more work (https://arxiv.org/pdf/1906.10742v2)

2. “Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning” — Raschka (2018) | arXiv preprint

The classic. A must-read by Sebastian Raschka, with many interesting takes: why simple accuracy metrics lie, the necessity of confidence intervals. Separates out performance estimation, model selection (same model, different hyperparameters), and algorithm selection (different ML algorithms). Goes into detail on bias, variance, different sampling techniques for model evals, as well as statistical tests for robust comparisons. (https://arxiv.org/pdf/1811.12808v3)

Phase 2: Shifting of the Boundary (How LLMs made things harder)

1. “A Survey on Evaluation of Large Language Models” — Chang et al. (2024) | ACM Transactions on Intelligent Systems and Technology (TIST)

Comprehensive survey covering LLM evaluation across multiple domains and tasks - what to evaluate, where to evaluate, and how to evaluate. The models covered may seem dated, but the concepts are still relevant. Covers tasks such as natural language understanding & generation, reasoning, robustness, bias and toxicity, trustworthiness. Outlines general vs. specific benchmarks; and automatic, human and crowd-sourced evaluations. Highlights gaps in bias, reasoning, memory, dynamic and robustness (which are still relevant today). Covers opportunities for AGI benchmarks, behavioral, robustness, dynamic, trustworthiness evals. (https://arxiv.org/pdf/2307.03109v9)

2. “On Robustness and Reliability of Benchmark-Based Evaluation of LLMs” — Lunardi et al. (2025) | arXiv preprint

A reminder that real-world applications involve significantly more variability than what standardized benchmarks can capture, and to be skeptical about the reliability of benchmark-based evaluations. Simple paraphrasing alone can easily lead to lower consistency, greater confusion. Data contamination makes benchmarks less reliable. Highlights the (semantic) brittleness of LLMs and the need to go beyond static benchmarks. (https://arxiv.org/pdf/2509.04013v1)

Phase 3: The Reality (When the Rubber Meets the Road)

Academics optimize for benchmarks; practitioners optimize for “doesn’t get us sued.” These guides bridge that gap.

1. “Your AI Product Needs Evals” — Hamel Husain | Blog post (hamel.dev)

Read this if you want a guide on moving from “vibe checks” to systematic evaluation. Proposes a 3-level hierarchy: Unit Tests (scoping tests, assertions, creating test cases) -> Human/Model Evals (via logs, human reviews, automated evals) -> A/B Testing (to see how it works in real world). Highlights the invaluable role of evals in iterating quickly (which I full agree with), and curating good testing data (which could be used for fine-tuning later). Key takeaway - start simple, break things down. (https://hamel.dev/blog/posts/evals/)

2. “Task-Specific LLM Evals that Do & Don’t Work” — Eugene Yan | Blog post (eugeneyan.com)

A critique of generic metrics. Shows why BLEU/ROUGE are not as useful for reasoning and provides specific alternatives for a range of tasks. Going beyond accuracy, using probability based metrics, using models to compare source and generated text, using learned metrics, checking reproduction of training data to address copyright risks, classifiers for toxicity detection were some key takeaways. (https://eugeneyan.com/writing/evals/)

Phase 4: The Experimental (Evaluating AI Agents & AI as a Judge)

1. “Evaluation and Benchmarking of LLM Agents: A Survey” — Mohammadi et al. (2025) | ACM SIGKDD 2025

A taxonomy for agent evaluation, distinguishing between objectives (behavior, capabilities, safety) and process (static vs. dynamic). Covers the “What” (assessing agent behaviour, capabilities, reliability, safety & alignment), the “How” (static testing with datasets, dynamic testing with simulations, benchmarks, code-based, LLM as judge and human in loop evals). Also interesting in covering enterprise hurdles - role-based access controls, reliability guarantees, long horizon performance, policy compliance. The taxonomy presented in Figure 1 is excellent. (https://arxiv.org/pdf/2507.21504)

2. “Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey” — Zhu et al. (2025) | arXiv preprint

Frames evaluation as an evolutionary process. 5 key aspects when looking at this evolution - complexity of environments, number of sources of instructions, dynamism of feedback, degree of multimodal perception, state of capabilities. Also looks at this from the perspective of environments (coding, web, OS, mobile, scientific, gaming), and capabilities (planning, self-reflection, interaction, memory). (https://arxiv.org/pdf/2506.11102)

3. “LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods” — Li et al. (2024) | arXiv preprint

Interesting survey that with interesting points in the areas of Functionality, Methodology, Application, Meta-Evaluation, and Limitations. Covers the role of LLM judges in response & model evaluation, reward modelling, as verifiers, annotation and generation of synthetic data. Goes into the use of LLM judges in single and multi LLM systems, e.g., using prompt engineering to guide judge, finetuning judges in single LLM systems; or using cooperation, communication or aggregation of LLM judges in multi-LLM systems. Reminder that LLM judges have the same issues - bias, vulnerability to adversarial attacks, recency bias, hallucinations. (https://arxiv.org/pdf/2412.05579v2)

4. “Agent-as-a-Judge: Evaluate Agents with Agents” — Zhuge et al. (2024) | ICML 2025

Argues that static LLM judges fail for agents because they miss the intermediate steps. Proposes Agent-as-a-Judge framework that extends LLM judges with additional capabilities with components that ask, graph, locate, read, search, retrieve, memory, plan. Finds that it can outperform humans and LLMs as judges. (https://arxiv.org/pdf/2410.10934v2)

The Standards

1. NIST AI Measurement and Evaluation Projects | Government resource

A selection of NIST AI measurement and evaluation related projects. (https://www.nist.gov/programs-projects/ai-measurement-and-evaluation/nist-ai-measurement-and-evaluation-projects)

2. Singapore’s AI Verify Global AI Assurance Pilot | Government resource

A selection of case studies on organizations partnering AI testing specialists. (https://assurance.aiverifyfoundation.sg/)

3. ISO/IEC TR 29119-11:2020 (Testing AI Systems) | International standard

Part 11: Guidelines on the testing of AI-based systems. (https://www.iso.org/obp/ui/en/#iso:std:iso-iec:tr:29119:-11:ed-1:v1:en)

“How do you know it works?”

It’s no longer a simple question with a simple answer. More and more, we need a portfolio of tools and techniques for evaluation and testing.

Any must-reads in this area that you would recommend?

#AIRiskManagement #AIEvaluation #AITesting #AIReadingList