When LLMs first became accessible and popular in 2022, one easy way to tell if someone was selling snakeoil was if they claimed that either ChatGPT or LLMs could forecast or predict something in the future. Someone who said that obviously did not understand how LLMs then worked, and were essentially hallucinating or bullshitting with great confidence.

LLMs are trained on text data. To forecast, you are working in the domain of time series data. Text and time series are both sequences, but they have fundamentally different characteristics.

Things have evolved since. Quite a lot actually. I would listen more patiently now for the details if someone said he or she used LLMs for forecasting due to shifts in the field. However, there is still a clear distinction between LLMs for language or text, compared to time series foundational models inspired by LLMs.

Let me explain this. In 4 short acts.

Act I: The Classical World of Time Series Modelling

For decades, way before neural networks were practically usable, forecasting or predictions with time series data was the domain of statisticians and econometricians. Time series data is fundamentally different from tabular, image or text data. You can shuffle rows in tables, mix up images, or rephrase text, and the meaning would still be largely intact.

The first, but not unique, property of time series is sequence. Mixing up the sequential order of time series data renders it meaningless. This is what it has in common with text.

This is the world of models like ARIMA. Understanding such models provides a clear understanding of what matters for time series data.

The AutoRegressive Integrated Moving Average (ARIMA) models and its variants dominated time series analysis for ages. They captured the essential insights needed to analyse or make predictions with time series data.

The core ideas:

• AutoRegressive (AR): Current values depend on previous values

• Integrated (I): Many series become predictable after differencing

• Moving Average (MA): Current values depend on errors of previous predictions

Conceptually, every time series could be understood as a combination of level, trend, seasonality, cycles and some noise. This is what makes it different from text.

Y = LEVEL + TREND + SEASONALITY + NOISE

Seasonal ARIMA (SARIMA) handled recurring seasonal patterns. Vector AutoRegression (VAR) tackled multiple related series. Holt-Winters exponential smoothing handled trends and seasons differently. For volatile financial series, Engle’s ARCH and Bollerslev’s GARCH gave us proper models for time-varying volatility - the technical term is conditional heteroskedasticity, but the everyday intuition is that the size of the typical move on a given day is itself a moving target. Harvey’s structural time series models decompose a series into latent components - level, trend, seasonal, cycle - each modelled as its own stochastic process, with the Kalman filter doing the inference behind the scenes. And modern decomposition extensions like MSTL (Multiple Seasonal-Trend decomposition using Loess) handle multiple seasonalities at once - think hourly demand with daily, weekly, and annual cycles all stacked together.

If you want to start playing with these classical models, R’s forecast package (auto.arima, ets, stl) is the easy on-ramp, and Python’s statsforecast from Nixtla gives you the same family of models. Meta’s Prophet popularised an additive-decomposition approach for forecasting and sits closer to the classical end of the spectrum despite being open-sourced by a deep-learning lab.

Another fundamental characteristic of time series that is critical is the degree of stationarity. Much more than text or images (words only change meaning slowly over time, same for images), time series in many domains continually evolve, and are what we call non-stationary.*

Recall ‘I’ for integration in ARIMA? That step leads to a differencing operation that allows the time series to be more stationary, and hence more predictable.

The classical world of time series forecasting, because of this focus on the underlying patterns of trends, seasons etc., was inherently explainable.

That’s Act I. But before I move on to Act II, I thought it would be useful to mention that forecasting is not the only task you can apply to time series. You can also nowcast (predict current unknown values with time series data to date, like GDP); classify time series patterns, detect outliers for time series, and so on and so forth. But the fundamental characteristics of time series data that need to be taken into account remain the same.

Act II: Machine and Deep Learning’s Struggles with Time

The power of machine learning models in the last decade meant many tried to use machine learning models such as support vector machines, random forests, and boosting tree models for time series forecasting. But these models did not fit naturally with time series data. Not that it could not work, but it was a hit and miss.

The natural question then was, why switch to these significantly more complex machine learning models when the classical models were good enough?

Then came advances in computer vision and natural language processing driven by deep learning. Computer scientists being computer scientists, they started looking for new domains to test these models on. Naturally, given the importance of time series data in many commercial and financial settings, computer scientists started using these models for different tasks on time series data.

But here’s the thing - you can’t just throw time series data at a regular deep learning model and expect magic. Remember the properties - sequence, patterns, stationarity?

Sequence models for natural language processing like Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks made a lot of sense. DA-RNN introduced dual-stage attention for input and temporal selection. Time-aware LSTMs handled irregular sampling via decay gates. Temporal Convolutional Networks showed that convolutional architectures could match or beat RNNs for many sequence tasks.

Answering the question of how to remember important information from way back in the sequence, while forgetting irrelevant noise applied equally to text and time series data.

There were many papers that focused on adapting RNNs, LSTMs, and even CNNs (designed for image data) to time series data. I would say the results were ambivalent. Sometimes you got fantastic results, sometimes a simple classical model would beat the complex deep learning model handily in a fraction of the time and computing power. A reviewer once asked me why a model I developed and trained for time series modelling was so large in size. It was only 12m parameters (miniscule compared to the billions and trillions for LLMs), but you get the idea. Probabilistic recurrent forecasters like DeepAR and deep state-space variants became quite popular. NeuralProphet - the DL successor to Meta’s Prophet, swapping the additive components for neural network blocks - sounded promising but empirically failed to beat plain ETS on standard benchmarks. The “DL doesn’t automatically win” pattern was already showing up.

So was it a pointless exercise? Not really.

One of the key differences between classical and deep learning time series models was that deep learning models could utilise multimodal time series data (time series of text, audio, sensor events, images, networks), whereas classical models were largely restricted to numerical time series.

There are also many other aspects of time series that we could focus on aside from sequence for time series, distinctly different from natural language or image data.

For example:

• Different time scales: Time series that change by the second, hourly, daily, quarterly and so on and so forth.

• Varying signal quality: Time series data that are inherently noisier than natural language or image data.

• Relationship dynamics: The importance of interactions between different time series.

There are many more. And deep learning models gave us a lot of flexibility to explore these characteristics.

In the early 2020s, even before ChatGPT, but after the seminal “Attention is All You Need” moment, many papers also explored the use of attention-based transformers for time series data.

It was a logical pairing. We use positional encodings to encode the position of tokens (or words) in transformers for text. Why couldn’t we do the same for time series steps?

A whole family came out of this - N-BEATS for interpretable basis expansion, the Temporal Fusion Transformer (TFT) for multi-horizon attention, Informer for long sequences, TS Transformer for self-supervised representation learning, Autoformer for decomposition + auto-correlation, and PatchTST. The PatchTST trick - that you can chop a series into patches and treat them like tokens - would turn out to be a conceptual seed for Act III. CoST pushed contrastive disentanglement of seasonal and trend representations.

In parallel, multimodal work fused text and prices for financial forecasting - event-driven stock prediction such as “Listening to Chaotic Whispers”, which added news attention. Fine-grained event typologies, stock embeddings from news + price history, hierarchical multi-task models for earnings-call volatility, REST’s relational event-driven framework, and FAST’s news-and-tweet time-aware network all extended this line.

I published several papers in this domain - using numerical and text time series with evolving networks. KECE combined knowledge graphs with numerical and textual time series. GLAM distinguished between global and local temporal patterns with adaptive curriculum learning to handle noise. GAME designed latent sequence encoders for multimodal data of different frequencies. DynMix used dynamic self-supervised learning with implicit and explicit network views, while DynScan learned slot concepts to handle non-stationary multimodal streams. These models showed strong results on financial forecasting, portfolio optimization, and ESG predictions, but they were highly specialized transformer models trained for specific tasks.

They were not general purpose or foundational models. Just using a transformer does not qualify. A general purpose or foundational model needs to be usable across different tasks.

But there were folks already researching these, even prior to ChatGPT coming onto the scene in 2022.

For readers who want to play with this era of models, the Python libraries darts (which also contains classical models), Nixtla’s neuralforecast, and HuggingFace’s transformer-based time-series pipelines are good entry points.

Act III: Foundation Models Learn the Language of Time

Interestingly, while I was researching multimodal time series models focused on networks, one of my PHD mates (Gerald Woo) was doing groundbreaking work on one of the first foundational models for time series data inspired by LLMs architectures.

This was the Moirai time series foundation model for universal forecasting. We attended each other’s research presentations once in a while, and I found his work fascinating. But I was already at the tail end of my PHD, so too late to switch tack.

Since then there have been many more foundational models for time series data inspired by LLMs architectures.

Now that you know all the special characteristics of time series, you would know that one cannot just throw a time series model into a transformer or an LLM and expect some magic to happen.

One key challenge was how to convert infinite numerical possibilities into a finite vocabulary that transformers can process. Language has a finite vocabulary, but not time series.

Different teams solved this differently:

Amazon’s Chronos quantizes continuous values into 4,096 discrete bins - essentially creating a “vocabulary” for time series. Google’s TimesFM treats time segments as “patches” like image processing. Salesforce’s MOIRAI uses multiple patch sizes for different temporal frequencies. There are other such models, but the fundamental issues being solved are similar. Address the tokenisation of time series, collate a large cross domain dataset, adjust the transformer architecture to address the unique characteristics of time series data.

In the months since the first version of this piece, the foundation-model wave has not slowed. But the field has clearly forked into two camps. Both are still very much “Act III” - they tokenize numerical time series and stick a transformer on top. They differ in where the parameters come from.

Camp A - Tokenize numbers, train a transformer from scratch.

This is the original recipe. The more recent successors mostly keep it and refine it:

• Chronos-2 - Amazon’s 2025 follow-up; adds in-context learning across related series and covariate-informed forecasting.

• ChronosX - extends Chronos to handle exogenous variables.

• MOIRAI-2 - simpler architecture, better data, generalises better than v1.

• MOIRAI-MoE - sparse mixture-of-experts variant; specialises across frequencies.

• Lag-Llama - open-source probabilistic TSFM.

• TimeGPT-1 - Nixtla’s offering.

Camp B - Don’t pretrain a TSFM. Reprogram a frozen LLM.

The intuition: LLMs already cost hundreds of millions to train. Maybe you don’t need to start from scratch - just teach the existing LLM to see numbers.

• Time-LLM - patches the series, reprograms each patch via cross-attention with text prototypes, freezes the LLM body, trains only the input projection and output head.

• LLMTime - encodes the series as a string of digits and lets the LLM autoregress. No training.

• GPT4TS / OFA - frozen GPT-2, fine-tune only the layer norms and positional embeddings.

• PatchInstruct - patch tokenisation + decomposition + neighbour augmentation as a prompting strategy.

What’s the point of foundational models for time series?

To me, the holy grail is probably few or zero-shot forecasting. Train once on massive time series datasets, then gain the ability to perform a range of tasks - forecast sales, detect equipment anomalies, or classify patterns across entirely new domains without additional training.

Still an unsolved problem I feel.

The key characteristics of time series data that we mentioned earlier are still key (such as sequence dependency, non-stationarity, varying frequencies, signal-to-noise ratios, and domain-specific patterns), and unlike text data, the nature of time series data in different domains (finance, healthcare, energy, retail, manufacturing, climate) can be vastly different and evolve significantly across time. We will talk a bit more about this below.

Act IV: The Agentic Turn - Forecasting as Reasoning

Acts I, II, and III all share an assumption. They assume forecasting is a single step problem. You feed in history, the model spits out the forecast or prediction.

The new wave has a few branches.

First, we can fuse different types of time series data at foundational model level across steps. From News to Forecast uses LLM agents to iteratively filter news, classify events by effect horizon, and fine-tune an LLM to emit digit sequences alongside a reflection loop.

Second, we can add reasoning. Training LLMs showed that reasoning at inference time - generating long chains of thought before producing an answer - improves performance on maths and code. Does it improve performance on time-series forecasting? TimeReasoner wraps series + timestamps + contextual features into a hybrid prompt, feeds it to a slow-thinking LLM, and explores three reasoning strategies - making the LLM think about the series before answering.

And finally, add multiple steps to the mix. AlphaCast - a training-free three-stage workflow (Investigator → Generator → Reflector) that mirrors how an expert forecaster works: prepare context, predict, critique, refine.

Conclusion

The only conclusion is that there is no conclusion yet. The jury is still out.

For a long time, a good place to look at the state of time series models were the Makridakis Competitions. M1-M3 (1980s-90s) were won by classical methods. M4 (2020) was won by Slawek Smyl’s hybrid ES-RNN - exponential smoothing married to LSTM - not by pure deep learning. M5 (2022) was won by LightGBM (gradient-boosted trees) with feature engineering, not deep architectures. M6 (2025) pushed into finance and reported that most teams underperformed simple benchmarks. The recurring lesson across forty years of competitions: the gains attributed to deep learning are usually gains from feature engineering, ensembling, or hybrid approaches. A example of a new leaderboard for foundation models is GIFT-Eval - 24 datasets over 144,000 time series and 177 million data points, spanning seven domains, 10 frequencies, multivariate inputs, and prediction lengths ranging from short to long-term forecasts.

But even when these foundation models top the leaderboards, the picture could be misleading.

Rethinking Evaluation in the Era of TSFMs found that benchmark scores are inflated - the test data often overlaps with the training data, either directly or because similar time periods appear in both. Strip the overlap out and the impressive numbers shrink.

Re(Visiting) TSFMs in Finance put the leading foundation models against decades of stock returns from markets around the world. The result: off the shelf, they did not beat ordinary baselines. Even fine-tuning didn’t close the gap. The only thing that worked was re-training the model from scratch on financial data - at which point you’ve essentially built a domain-specific model, not used a “foundation” one.

Multimodal and agentic forecasting are even harder to judge. Rethinking Multimodal TSF Evaluation points out that many “text helps forecasting” benchmarks are flawed - for example, the news used in testing has often already leaked into the model’s training data.

And that’s where we are now. Lots of progress, but still many open questions and uncertainties.

References

Act I - Classical time-series models, surveys, and competitions

• Forecasting: Principles and Practice - https://otexts.com/fpp3/ - Modern, accessible, free treatment of the whole classical lineage. Best starting point.

• Automatic Time Series Forecasting: The forecast Package for R - https://www.jstatsoft.org/v027/i03 - *Origin of auto.arima.

• Forecasting Seasonals and Trends by Exponentially Weighted Moving Averages - https://www.sciencedirect.com/science/article/abs/pii/S0169207003001134

• Exponential Smoothing: The State of the Art - Part II - https://www.bauer.uh.edu/egardner/3301H%20Operations%20Management/ESG%20Publications/2006%20Exp.%20Sm.%20State%20of%20the%20art%20-%20Part%20II.pdf.

• Macroeconomics and Reality - https://www.pauldeng.com/pdf/Sims%20macroeconomics%20and%20reality.pdf

• Autoregressive Conditional Heteroskedasticity with Estimates of the Variance of UK Inflation - https://www.jstor.org/stable/1912773

• The Story of GARCH: A Personal Odyssey - https://public.econ.duke.edu/~boller/Papers/GARCH_JoE_2023.pdf .

• Forecasting, Structural Time Series Models and the Kalman Filter - https://www.cambridge.org/core/books/forecasting-structural-time-series-models-and-the-kalman-filter/CE5E112570A56960601760E786A5E631

• MSTL: A Seasonal-Trend Decomposition Algorithm for Time Series with Multiple Seasonal Patterns - https://arxiv.org/abs/2107.13462

• StatsForecast / Nixtla - https://github.com/Nixtla/statsforecast

• The M4 Competition: 100,000 time series and 61 forecasting methods - https://www.sciencedirect.com/science/article/abs/pii/S0169207019301128

• The M5 Accuracy Competition: Results, Findings and Conclusions - https://www.sciencedirect.com/science/article/pii/S0169207021001874

• Forecasting with Gradient Boosted Trees: M5 Uncertainty Winner - https://www.sciencedirect.com/science/article/abs/pii/S0169207021002090

• The M6 Forecasting Competition: Bridging the Gap between Forecasting and Investment Decisions - https://arxiv.org/abs/2310.13357

Act II - RNN / LSTM / Transformer-era and multimodal cluster

• DA-RNN: Dual-Stage Attention-Based RNN - https://arxiv.org/abs/1704.02971

• Patient Subtyping via Time-Aware LSTM Networks (T-LSTM) - https://dl.acm.org/doi/10.1145/3097983.3097997

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (TCN) - https://arxiv.org/abs/1803.01271

• DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks - https://arxiv.org/abs/1704.04110

• Deep State Space Models for Time Series Forecasting - https://papers.nips.cc/paper/2018/hash/5cf68969fb67aa6082363a6d4e6468e2-Abstract.html

• NeuralProphet: Explainable Forecasting at Scale - https://arxiv.org/abs/2111.15397

• A Hybrid Method of Exponential Smoothing and Recurrent Neural Networks for Time Series Forecasting (ES-RNN, M4 winner) - https://www.sciencedirect.com/science/article/abs/pii/S0169207019301153

• N-BEATS: Neural Basis Expansion Analysis for Interpretable Time Series Forecasting - https://arxiv.org/abs/1905.10437

• Temporal Fusion Transformers for Interpretable Multi-Horizon Time Series Forecasting (TFT) - https://arxiv.org/abs/1912.09363

• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting - https://arxiv.org/abs/2012.07436

• A Transformer-based Framework for Multivariate Time Series Representation Learning - https://arxiv.org/abs/2010.02803

• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (PatchTST) - https://arxiv.org/abs/2211.14730

• CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting - https://arxiv.org/abs/2202.01575

• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting - https://arxiv.org/abs/2106.13008

• Deep Learning for Event-Driven Stock Prediction - https://www.ijcai.org/Proceedings/15/Papers/329.pdf

• Listening to Chaotic Whispers: A Deep Learning Framework for News-Oriented Stock Trend Prediction - https://arxiv.org/abs/1712.02136

• Incorporating Fine-grained Events in Stock Movement Prediction - https://aclanthology.org/D19-5106/

• Stock Embeddings Acquired from News Articles and Price History, and an Application to Portfolio Optimization - https://aclanthology.org/2020.acl-main.307/

• HTML: Hierarchical Transformer-based Multi-task Learning for Volatility Prediction - https://dl.acm.org/doi/10.1145/3366423.3380128

• REST: Relational Event-driven Stock Trend Forecasting - https://arxiv.org/abs/2102.07372

• FAST: Financial News and Tweet Based Time Aware Network for Stock Trading - https://aclanthology.org/2021.eacl-main.185/

• Learning Knowledge-Enriched Company Embeddings for Investment Management - https://dl.acm.org/doi/abs/10.1145/3490354.3494390

• Investment and Risk Management with Online News and Heterogeneous Networks - https://dl.acm.org/doi/full/10.1145/3532858

• Guided Attention Multimodal Multitask Financial Forecasting - https://aclanthology.org/2022.acl-long.437/

• Dynamic Multimodal Implicit and Explicit Networks for Multiple Financial Tasks - https://ieeexplore.ieee.org/abstract/document/10020722/

• Dynamic Multimodal Slot Concepts from the Web - https://dl.acm.org/doi/full/10.1145/3663674

Act III - Foundation models (Camp A: train-from-scratch TSFMs)

• Unified Training of Universal Time Series Forecasting Transformers (MOIRAI) - https://arxiv.org/abs/2402.02592

• Chronos: Learning the Language of Time Series - https://arxiv.org/abs/2403.07815

• A Decoder-Only Foundation Model for Time-Series Forecasting (TimesFM) - https://research.google/blog/a-decoder-only-foundation-model-for-time-series-forecasting/

• Chronos-2: From Univariate to Universal Forecasting - https://arxiv.org/abs/2510.15821

• ChronosX: Extending Time-Series Foundation Models to Support Exogenous Variables -

AI Horizon Forecast

ChronosX: Extending Time-Series Foundation Models to Support Exogenous Variables

Foundation models excel in univariate time-series benchmarks…

Read more

a year ago · 6 likes · Nikos Kafritsas

• MOIRAI 2.0: When Less Is More for Time Series Forecasting - https://arxiv.org/abs/2511.11698

• MOIRAI-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts - https://arxiv.org/abs/2410.10469

• Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting - https://arxiv.org/abs/2310.08278

• TimeGPT-1 - https://arxiv.org/abs/2310.03589

Act III - Foundation models (Camp B: LLM-reprogramming)

• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models - https://arxiv.org/abs/2310.01728

• Large Language Models Are Zero-Shot Time Series Forecasters (LLMTime) - https://arxiv.org/abs/2310.07820

• One Fits All: Power General Time Series Analysis by Pretrained LM (GPT4TS / OFA) - https://arxiv.org/abs/2302.11939

• PatchInstruct: Forecasting Time Series with LLMs via Patch-Based Prompting and Decomposition - https://arxiv.org/abs/2506.12953

Act IV - Multimodal text + TS

• From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection - https://arxiv.org/abs/2409.17515

• Unlocking the Value of Text: Event-Driven Reasoning and Multi-Level Alignment for Time Series Forecasting - https://arxiv.org/abs/2603.15452

Act IV - Agentic & reasoning frameworks

• AlphaCast: A Human Wisdom-LLM Intelligence Co-Reasoning Framework for Interactive Time Series Forecasting - https://arxiv.org/abs/2511.08947

• Empowering Time Series Forecasting with LLM-Agents (DCATS) - https://arxiv.org/abs/2508.04231

• Can Slow-Thinking LLMs Reason Over Time? Empirical Studies in Time Series Forecasting (TimeReasoner) - https://arxiv.org/abs/2505.24511

Evaluation

• GIFT-Eval: A Benchmark for General Time Series Forecasting Model Evaluation - https://arxiv.org/abs/2410.10393 · leaderboard at https://tsfm.ai/benchmarks/gift-eval

• Rethinking Evaluation in the Era of Time Series Foundation Models: (Un)known Information Leakage Challenges - https://arxiv.org/abs/2510.13654

• Rethinking Multimodal Time-Series Forecasting Evaluation - https://openreview.net/forum?id=Z1TMV4bGuu

• Re(Visiting) Time Series Foundation Models in Finance - https://arxiv.org/abs/2511.18578

Starting with models. My favourite.

The first models I ever met were simple curves. Bootstrapping zero rates off swap quotes in a Masters of Financial Engineering classroom. Then the more sophisticated cousins, Black-Scholes, SABR, Hull-White, Heston. I still remember the books by Emanuel Derman, Massimo Morini, and many others. For many years, the SR 11-7 was the only guidance governing this world. Though you could see snippets of it in parts of the Basel capital rules. To be honest, I was not a great student and was confused about such quantitative models for many years.

Then I met machine learning models. Decision trees. Random forests. Gradient boosted trees. Support vector machines. Then deep learning models. RNNs, CNNs. Still models, but slightly more generalized. To be honest, it was only when I was learning machine and deep learning that I finally understood the quantitative models I learnt earlier. Same for LLMs. And agents are just systems wrapped around LLMs.

Once you see the main parts - the selected function, the objective, the optimization method, and the evaluation method - you realize that they ain’t very different. Asking questions about any model ultimately comes down to these parts, and the assumptions underlying the model.

And it’s the AI model that ultimately makes an AI system different from any other software system. It’s also why in 2024, we did a review of AI model risk management in banks, when I was still in MAS. If there was any place to start, looking at the models was as good as any.

And it’s where the 3 ‘U’s arise.

Uncertainty. All models have it. Irreducible randomness you can’t eliminate. Reducible gaps you can close with more data. But always some left.

Unexpectedness. More complex models behave in ways nobody designed. Emergent capabilities. Adversarial vulnerabilities. Gaming the system. Misalignment.

Unexplainability. The degree to which we can explain a decision varies. Transparency, explainability, interpretability, but ultimately about understanding.

The Fed, OCC and FDIC dropped SR 26-2 on April 17. Replaced SR 11-7 after fifteen years. Buried in it, a single-sentence carve-out: ‘GenAI and agentic AI are explicitly excluded from scope.

For me personally, given the connections between models, whether quantitative or LLM, this exclusion puzzles me a little.

What do you think?

#ThinkingInSpectrum #ModelRisk #AIRiskManagement

I’ve been doing a number of talks on the intersection of two disciplines.

First discipline. Model risk management. My model risk supervision days.

Second discipline. Computer science. AI models. My ill-advised side quest to get a PhD in computer science at 42.

The MAS AI risk management guidelines I wrote from these covers 3 units of governance that relate to these disciplines - model, system, use case. They form a good foundation. But I feel like something has shifted.

1️⃣ Model - Obviously the domain of model risk management (MRM). The world of the late SR 11-7 & the new SR 26-2 from the US. As well as the UK’s SS 1/23 and Canada’s E23. What OpenAI, Anthropic etc. release and spend shitloads of money on.

2️⃣ System - The domain of technology risk management. Too many frameworks here to list. But the recent “AI” vulnerabilities that have been in the news, from Anthropic’s troubles to Vercel have nothing to do with the AI model, but the fragility of systems in this rush to do “AI”.

3️⃣ Use Case · What AI risk management focuses on. Anchored by NIST AI RMF, ISO 42001. MAS’ AI risk management guidelines. This is closest to what we experience when we use AI, but is also the least scalable.

And above the line.

4️⃣ Archetype. Prior to Agentic AI, I already saw folks grouping Generative AI into archetypes - question and answer, summarization, retrieval augmented generation. LLMs are general purpose tools, but grouping them helps with scalability. For once stereotyping things helps.

5️⃣ Capability. Bounded class of actions with authority, constraints, evidence requirements. Capabilities extend the idea of archetypes to the world of Agentic AI. Where we decompose and compose what AI agents do. See the paper on Agentic MRM by Lukasz Szpruch, Agus Sudjianto, Tanveer Bhatti and me on this; and the Agentic Risk and Capabilities (ARC framework) by Shaun Khoo and Roy Ka-Wei Lee.

6️⃣ Workflow. I know terms like trajectories and runtime have come into vogue (runtime is the term we use in the title of the Agentic MRM paper). But I feel like they are manifestations of what happens when AI agents do something, rather than a unit of governance. Workflows seem closer to something we can govern. And they relate nicely to the idea of the action space x autonomy perspective of AI agents (see Singapore’s Model Governance Framework on Agentic AI). As you relax the constraints of workflows, you get trajectories and runtimes that can go wider and wilder.

The three units above the line represent a new set of risks. But somehow I feel they also represent a new route to scalability.

Models, systems and use cases are usually specific. Whereas archetypes, capabilities and workflows can be generalized across a range of applications.

Just thinking out loud. Let me know your perspectives. I’ll try to write more about this in the weeks ahead.

#AIGovernance #AgenticAI

The first part, “My Shifu“, covered how I met Johnny Lau through a crazy experiment called Creative Youth Xchange, and his view that technology forces artistic purity rather than threatening it.

I’ve been sitting on this, but this post by Arvind Narayanan on how creativity in art and creativity in science are different made me get back to this second part.

This article goes deeper into the creative process. About creative production, and why AI may not be that great a leap.

Is AI Really A Leap?

One of the key fears of creatives is that this leap into AI may be one too far. Further than the camera. Further than digital tools.

Something that creativity cannot keep up with. But is it realistic to think that all art in the future will be purely from AI?

So I asked Johnny about his jumps from pen-and-paper comics to different technologies, the most recent being a Singlish speaking robot. Most people would call that a transformation. He didn’t:

“It was never a leap for me. Right from the beginning when I launched the first book in 1990, I engineered a range of products such as t-shirts & mugs, a 13-inch ruler and bumper sticker with a tagline: Beware of Kiasu Driver. The latter became a bestseller instantly.”

My read. This leap is perhaps not that different from the ones in the past for creatives. And AI is perhaps more of an opportunity than a threat.

“For me it has always been a concept, not a book nor a sticker ... My brain doesn’t function the way where sectors and industries are differentiated. I had to develop a language to align myself with how the world functions.”

A concept. Not a product. Not a comic.

Here’s what I think is going on. Narayanan’s distinction is useful, creativity in art is emotional, social, driven by taste. Creativity in science is instrumental, evaluable, a systematic search through possibilities. AI is getting better at the second kind. The first is harder to touch.

Johnny’s concept is the first kind. Not the drawing - the reason for the drawing. The cultural nerve. The recognition that makes a Singaporean laugh at their own kiasu-ness. That doesn’t live in a dataset. It comes from being someone, from somewhere.

The production - t-shirts, mugs, strips - is the second kind. Patterns. Systems. Things that can be decomposed and delegated.

Johnny worked this out decades ago. He kept the concept. He systematized the production. And the person who showed him how wasn’t an AI researcher. It was Jim Davis.

The Garfield Revelation

I asked about architecture at USC. He said something I didn’t expect - that USC was never about architecture for him:

“The years at USC were never about the study of building design. I took full liberty to immerse myself in the world of media and entertainment when I landed in Los Angeles.”

That’s where he met Garfield. Not the cat. The operation.

“I was stunned by Davis and how he had built a team of artists and writers to produce daily comics strips for the lasagna loving cat for its national syndicate. He had deployed an animation studio system to run his comics studio.”

Here’s what he described. Davis ran his comics like a factory. Team meeting on Monday. Different themes tossed around for the week’s seven strips. Once decided, a lead artist carried out the drafts. The process cascaded through the team. And at the end:

“Davis merely just drew Garfield’s eyes as a gesture of completion.”

Garfield’s eyes. The signature. Everything else was system. That one stroke was authorship.

I sat with that image for a while.

Johnny brought this model back to Asia. Most Asian comics were produced by a single artist - one person doing everything. The Garfield team showed him a different path. It became Comix Factory, the studio that produced Mr Kiasu.

Agents and Art

Here’s the connection I keep thinking about.

Davis’s studio system was, in effect, an early version of what AI promises every creator today. Break the work into steps. Specialize each step. Let the system handle volume while the creator handles vision.

I’ve written about agentic AI before - systems where multiple AI agents each handle a different part of a task. One agent plans, another retrieves information, another reasons, another generates output. The whole thing works because each part has a defined role and you can observe what each part does.

Davis was essentially running an agentic system in the 1980s. Just with humans instead of models. One person for drafts, another for inking, the process cascading through the team. And the orchestrator - Davis - only needed to draw the eyes.

Davis understood in the 1980s what the AI industry is selling in the 2020s: you don’t need to do everything yourself. You need to know what only you can do.

For Davis, it was the eyes. For Johnny, it’s the concept.

And Johnny made the connection himself:

“Similarly with AI, we just have to change our mindset on the method of production. The creation portion doesn’t change as it will always come from the depth of our minds.”

Change the method of production. Keep the creation. That’s not a theory from someone who’s read a few articles about AI. That’s a principle from someone who’s been decomposing creative labor since 1990.

A nice footnote: in 2023, Mr Kiasu and Garfield collaborated on a joint library programme with Singapore’s National Library. A one-of-a-kind global collaboration between two comics characters - the Singaporean one modeled on the American one’s production system - sharing a stage forty years later.

AI As Just Another Method

There’s a broader pattern here that I think matters for anyone thinking about AI and creative work.

We tend to frame the AI question as: will machines replace artists? That’s the wrong question. The better question is: what part of your creative work is system, and what part is signature?

If you’re a writer, maybe the system is research, outlining, drafting. The signature is voice, judgment, the instinct for when a sentence lands. If you’re a designer, maybe the system is rendering, iteration, production. The signature is the concept that no brief could have specified.

Davis knew. He drew the eyes and let the system do the rest. Not because he was lazy. Because he understood what only he could do.

AI doesn’t change this question. It just makes it unavoidable.

Johnny figured this out before AI existed. He’s been running this model for thirty-five years - concept first, production second, and the concept crosses every medium, every platform, every technology that comes along.

The question I’m leaving with you: what’s the part that only you draw?

More soon.

Yesterday, there was news about a threat actor compromising the popular LiteLLM package on PyPI. litellm is an ubiquitous library that many teams use as a unified interface to call different LLM APIs. The attacker didn’t need anything sophisticated. They poisoned the CI/CD pipeline through a compromised security scanner (the irony!), then pushed backdoored versions of litellm that harvested SSH keys, cloud credentials, and secrets on every Python startup. Seems like the hackers are now working hard - going through a 300gb treasure trove and extorting multi-billion-dollar companies.

This is what third-party AI risk actually looks like. Not the dramatic scenarios we imagine - collusion, rogue AI, coordinated manipulation. Sometimes it’s a simple file in a package you installed last Tuesday.

And it’s not just open-source libraries. Every AI vendor you use sits on its own supply chain of dependencies, sub-processors, and infrastructure providers. A single compromised link and you inherit the risk.

So how do you govern AI you don’t control? And maybe don’t even know exists.

I can’t say I have the answer. But I can offer a reading list.

Note: I have used open-access links from arXiv as far as possible.

Where and Why Does It Break?

1. “AI Auditing: The Broken Bus on the Road to AI Accountability” - Birhane et al. | IEEE SaTML (2024)

Taxonomizes current AI audit practices across regulators, law firms, civil society, journalism, academia, and consulting. Finds that only a subset of AI audit studies translate to desired accountability outcomes. The title says it all - we’re doing audits, but many of them aren’t actually getting us where we need to go. 📄 arXiv:2401.14462

2. “Dislocated Accountabilities in the ‘AI Supply Chain’: Modularity and Developers’ Notions of Responsibility” - Widder & Nafus | Big Data & Society (2023)

Developers building AI from preexisting modules often believe responsible AI belongs to “the next or previous person in the imagined supply chain.” Everyone assumes someone else is managing the risk. 📄 arXiv:2209.09780

3. “Understanding Accountability in Algorithmic Supply Chains” - Cobbe, Veale & Singh | FAccT (2023)

Explores how algorithmic supply chains create distributed responsibility and limited visibility due to what the authors call the “accountability horizon” - you can only see so far. Also covers cross-border supply chains and regulatory arbitrage, which matters when your AI vendor operates across jurisdictions with different rules. 📄 arXiv:2304.14749

How can we perhaps solve it?

1. “AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations” - Stosz et al. | AI Evaluation Foundation (2025)

A voluntary standard and checklist that defines what evaluators actually need from AI providers: independence from the provider, sufficient access to assess characteristics of interest, and transparency in sharing methods and findings. The gap between what this standard says you need and what most vendors actually give you is the governance gap. 🔗 AI Evaluation Foundation

2. “Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance” - Raji et al. | AAAI/ACM AIES (2022)

Synthesizes lessons from financial, environmental, and health regulation on crafting effective external oversight systems. The key insight for me: audits alone won’t achieve accountability. You need deliberate design and institutional weight, the same lesson financial regulators learned after Enron. 📄 arXiv:2206.04737

3. “Model Cards for Model Reporting” - Mitchell et al. | FAccT (2019)

The foundational paper proposing that released models be accompanied by documentation detailing their performance characteristics. This paper is from 2019. It’s now 2026. Notice how few vendors actually provide this level of transparency. In fact, some studies have shown that such transparency is declining, rather than increasing due to greater awareness. 📄 arXiv:1810.03993

4. “Third-party compliance reviews for frontier AI safety frameworks” - Homewood et al. | arXiv preprint (2025)

Explores third-party compliance reviews where an independent external party assesses whether a frontier AI company complies with its own safety framework. Discusses benefits (increased compliance, assurance to stakeholders) and real challenges (information security risks, cost burdens, reputational damage from findings). This is the emerging infrastructure for keeping vendors accountable - but it’s still nascent. 📄 arXiv:2505.01643

5. “Implementing AI Bill of Materials (AI BOM) with SPDX 3.0” - Bennet et al. | Linux Foundation Research (2025)

Extends the Software Bill of Materials concept to AI, including documentation of algorithms, data collection methods, frameworks, licensing, and compliance. If you want to know what’s actually inside the AI system you’re buying - and what changes when the vendor updates it - this is the direction. Think of it as the AI equivalent of ingredient labelling. 📄 arXiv:2504.16743

6. “AgentFacts: Universal KYA Standard for Verified AI Agent Metadata & Deployment” - Grogan | arXiv preprint (2025)

Forward-looking. Proposes a “Know Your Agent” standard with cryptographically-signed capability declarations and multi-authority validation. If something like this existed at scale, it would reduce the custom integration friction that makes switching so expensive. We’re not there yet, but this is the direction things need to go. 📄 arXiv:2506.13794

7. “Consultation Paper on Proposed Guidelines on Third-Party Risk Management” - Monetary Authority of Singapore | MAS (March 2026)

And last but not least. MAS just released this in March 2026. Supersedes the old outsourcing guidelines and extends expectations to all third-party arrangements, not just outsourced services. Covers risk assessment, due diligence, contracting, onboarding, ongoing monitoring, and termination. Requires FIs to maintain a register of third-party arrangements, monitor concentration risk, and extend oversight to sub-contractors. AI appears in a footnote, where it refers to MAS AI Risk Management Guidelines I wrote, but the relevance of these guidelines to AI is clear. 🔗 MAS Consultation Paper

I think this is one of the hardest problems to solve in AI risk management. Because it requires everyone to work together. And that’s really hard today.

#ThirdPartyAI #AIRiskManagement #VendorManagement #AIGovernance #AIReadingList

I always thought it was kind of stupid. Since shouting it was asking for the bullets to be aimed at you. I always felt I was lucky to move from infantry to be a scout. Scouts never ran towards bullets.

But I think I understand the need for it more now. Shouting it in the army helps make people aware of the gap that you are moving towards.

In one of my past weekly reflections on LinkedIn, I said three gaps kept surfacing. The common language gap. The contextualisation gap. The last-mile gap. Same need for awareness.

Frameworks exist. Guidelines exist. I wrote some of them. But translating frameworks into something relevant for one’s context is not easy. Especially for individuals and smaller firms.

Last week, MAS launched the MindForge AI Risk Management Toolkit. I was deeply involved as it needed to be aligned to the MAS AI Risk Management Guidelines I wrote.

The MindForge AI Risk Management Toolkit is an operationalisation handbook with case studies. It’s an attempt at bridging the messy middle I keep writing about. But I suspect, aside from large financial institutions, it may still be a lot to chew on for everyone else.

Last week, a friend who leads manpower development also asked me something similar: what about everyone else?

This is my early attempt at an answer.

GOT, not Game of Thrones, but an AI Governance Tool.

At govern.simplyboring.ai.

Describe what you are using AI for. Get a tailored governance pack in minutes. The standards that apply to your situation. The controls to put in place. How to implement them. What evidence to prepare.

Built on public information. The MAS AI Risk Management Guidelines I wrote. The MindForge toolkit that just launched. The US Financial Services AI Risk Management Framework. The EU AI Act. NIST AI RMF. ISO 42001. And other international frameworks. Cross-referenced and connected. And a little of my own perspectives.

The AIRG tells you what to do. MindForge shows how the industry is doing it. GOT asks what you specifically are doing, and tells you what applies to you.

Not just for the large institution with a full second line (and third line) of defence. For everyone else. Individuals. Small firms.

It’s an early prototype. It will almost certainly have errors. And a little slow - around 45-60 seconds for each report. And I think it’s still a little iffy for individuals.

But I tried to ground it in real frameworks and design it for real situations so anyone, not just banks.

Please try it. Tell me what’s missing. Tell me what’s wrong.

#Mindforge #NIST #AIRiskManagement #SimplyBoringAI #Grounding #AIRG

I don’t like OpenClaw. But pretty hard to resist the urge to try.

I think it does demonstrate something interesting about agents - 1) the additional utility when you hook it to a heartbeat to do something at a set frequency, or cron for a scheduled task; 2) the convenience of letting it communicate with you over a messaging app.

But I really can’t see what else it offers in additional to say Claude Code, which already allows for multi-agent workflows.

Ok, maybe if being exposed to the internet, or losing your data is your thing, then I agree, OpenClaw without any additional safeguards is definitely for you.

OpenClaw’s issues must be obvious. NVIDIA just announced NemoClaw at GTC - essentially OpenClaw with enterprise-grade security guardrails baked in. NanoClaw, a lightweight alternative, takes a different approach by isolating each agent in its own Docker container.

And so when I tried it, I made some adjustments to help mitigate this. Probably not perfect, but I think I can live with its risks with these controls. And I think you can use them for any of the other Claws too.

So here’s the short guide on what I did. I won’t copy and paste all the scripts. Just use my text below as a prompt and I am sure any of the LLMs can help you.

The Setup

I run it on a DigitalOcean VPS - Ubuntu 24.04, 2 vCPU, 2GB RAM, $12/month, Singapore region. I went with a manual setup instead of the 1-Click app because I wanted full control over every layer of the stack.

The finished stack looks like this:

What I use it for: OpenClaw runs 24/7 on the VPS. Every hour it wakes up, checks my task files, and stays silent if nothing is urgent. Every morning it sends me a briefing on Telegram. Claude Code runs locally for deep work and pushes outputs into a shared AGENT-INBOX folder. OpenClaw picks them up.

Total cost: $12/month for the server. OpenClaw lets you plug in your own API key. There’s an option for using your existing Claude key that will come up after the OpenClaw onboard.

Now, on to the controls.

Control 1: Server Hardening

Out of the box, a VPS is exposed. Default root login, password authentication, every port open. The first thing to do is reduce the attack surface.

Every service you run is an attack surface. Make each one as small as possible, and add friction at every layer so that even if one layer fails, the next one holds.

The key steps: update everything, create a non-root user (don’t run your server as Administrator), disable password authentication so only SSH keys can log in, and restrict root login entirely.

I won’t paste every command here - ask Claude or ChatGPT to give you the steps for hardening a server for Ubuntu 24.04 and it’ll give you the full thing.

Control 2: Firewall and Fail2ban

Two layers. UFW (Uncomplicated Firewall) controls who can knock on your door at all - deny all incoming by default, then explicitly allow only what you need. Fail2ban monitors your login logs and automatically bans IPs that fail authentication too many times.

One thing to watch out for: enable UFW after you allow SSH, not before. Get the order wrong and you immediately lose access to your own server.

But even with both of these, your SSH port is still visible to the public internet. Bots still find it, still try it. Fail2ban is a mitigation, not a solution.

Control 3: Tailscale - Making the Server Invisible

What if the SSH port wasn’t visible at all?

Tailscale creates a private encrypted network across the internet - but only devices you authenticate can join. Think of it as a private LAN that ignores physical location. Every device gets a private IP in the 100.x.x.x range. Your VPS gets one. Your laptop gets one. They talk through an encrypted WireGuard tunnel regardless of where they physically are.

The key part: your VPS’s public IP can have all ports closed. Tailscale establishes peer-to-peer through NAT traversal - no open ports required. Install it on the VPS, install it on your laptop, log in with the same account - done.

Once Tailscale is working, update your UFW rules to allow SSH only on the Tailscale interface, and remove the public SSH rule. After this, run a port scan on your VPS’s public IP. You’ll see nothing. The server is invisible. Fail2ban becomes a backup layer rather than the first line of defence.

As always, test SSH via the Tailscale IP in a second terminal before removing public access.

Installing OpenClaw

With the controls in place, this is the straightforward part.

curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash - 

sudo apt install -y nodejs

sudo npm install -g openclaw

openclaw onboard

For NemoClaw, it will probably look like this (based on Nvidia’s instructions)

curl -fsSL https://nvidia.com/nemoclaw.sh | bash

nemoclaw onboard

The setup wizard walks you through AI provider (I use Anthropic), API key, and gateway token. It installs OpenClaw as a user-level systemd service - not a system-level one. This matters:

This works:

systemctl --user restart openclaw-gateway

This silently fails:

systemctl restart openclaw-gateway

Every OpenClaw service command needs --user. Burned time on this.

The dashboard

OpenClaw has a web dashboard. It binds to localhost only, so you access it via SSH tunnel:

openclaw dashboard    # prints URL with token

ssh -N -L 18789:127.0.0.1:18789 openclaw@YOUR_TAILSCALE_IP

Then open the full URL including the #token= fragment in your browser.

Don’t open

http://localhost:18789/

bare. The #token= fragment is the authentication handshake. Without it, the gateway sees an unapproved device and shows “pairing required.” The dashboard looks like it loaded - it hasn’t connected. Use the full URL from openclaw dashboard.

Syncthing - The Sync Layer

OpenClaw on the VPS needs a way to receive tasks and send outputs back. I use Syncthing - open-source, peer-to-peer file sync, no cloud intermediary.

The synced folder (AGENT-INBOX) lives inside my Obsidian vault on Windows and mirrors to remote VPS. I write tasks in Obsidian, they appear on the VPS within seconds. OpenClaw writes outputs on the VPS, they appear in Obsidian within seconds.

One thing to watch out for: install Syncthing from the official apt repository, not Ubuntu’s default packages - the default version is outdated enough to cause silent sync failures when paired with a newer Windows client.

The Heartbeat

The heartbeat is the reason I tried OpenClaw in the first place. It’s the cron-like behavior - wake up on a schedule, check your files, act without being asked.

openclaw config set agents.defaults.heartbeat.every 1h

openclaw config set agents.defaults.heartbeat.target last

openclaw config set agents.defaults.heartbeat.activeHours.start "06:00"

openclaw config set agents.defaults.heartbeat.activeHours.end "01:00"

openclaw config set env.vars.TZ "Asia/Singapore"

Note: use the full dotpath agents.defaults.heartbeat.every, not just heartbeat.every. The docs show JSON structure, the CLI uses dotpaths.

If OpenClaw finds nothing urgent, it stays silent. No Telegram noise. Only substantive findings surface. This is controlled by a HEARTBEAT.md in your workspace - a checklist the agent follows on every run.

One important step after setup: delete the BOOTSTRAP.md file in your workspace. It’s OpenClaw’s first-run onboarding wizard that blocks the agent on every gateway start until a human responds. Once you’ve completed setup, remove it or the agent stays stuck in bootstrapping state indefinitely.

The Security Audit

OpenClaw has a built-in security audit you can run with openclaw security audit --deep. Worth running after setup to confirm your controls are in place.

The most important thing it tells you is the trust model. OpenClaw operates as a “personal assistant” - single user, single trusted operator. It’s not designed for multiple users sharing one gateway. If you’re thinking about running this for a team or clients, that’s the line that should make you pause.

Is It Worth It?

Honestly, I’m still not sure.

The heartbeat and the Telegram integration are genuinely useful - having an agent check on things hourly and surface only what matters is different from having a chatbot you talk to. And the data sovereignty matters to me.

I’ll keep running OpenClaw for a while to see if it sticks. See if the habit of an always-on agent changes how I work. But I wouldn’t tell anyone to just run it.

If I were starting today, I’d probably look at NanoClaw first - it’s ~4,000 lines of code versus OpenClaw’s > 400,000, and agent gets its own isolated container. NemoClaw is worth trying too, especially if you’re in an enterprise context. Both are open source.

But the manual controls I described above? They’d apply to any of these. Server hardening, Tailscale, firewall rules - that’s not OpenClaw-specific.

If you’ve tried self-hosting OpenClaw (or variants such as NanoClaw), what additional controls did you put in place? Also what else do you find useful about it (beyond say Claude Code or Codex) that warrants the effort?

#AI #AgenticAI #Security #OpenClaw #NemoClaw

Left Panel: VICReg

Self-supervised training in AI just means training without labels. We use it in LLMs. Pretraining a model by masking parts of inputs and getting it to predict the masked parts. One issue in self-supervised learning is collapse. AI likes to find shortcuts. And when it does, everything converges to the mean.

Most methods fight this with training or architectural tricks. VICReg uses just three interesting loss terms:
➡️ Variance so that there’s always some differences between the representations it learns.
➡️ Invariance so that the same input gives the same representation.
➡️ Covariance so that each dimension learns something different

Invariance is the main learning objective. Variance + covariance are the regularisers that prevent it from collapsing.

I liked VICReg’s idea. So I added a twist for a paper I published on a model called DynMIX. I fed it two fundamentally different views of the same company - explicit (constructed) and implicit (learnt from data). Another twist. VICReg's variance target is static. For financial data, that made no sense. Markets are volatile so variance can’t be static. So I made the target dynamic, based on variances of stock market returns.

Right Panel: JEPA

The core of the paper by Yann LeCun - JEPA - takes a different approach to the same problem. Instead of just learning representations, it predicts them. In AI, we usually predict something real - a label (dog, cat), or a number (stock price). JEPA predicts the underlying representation instead. Think of it as the model learning to imagine what something looks like in its own internal language, before it's seen in the real world.

Three components:
➡️ Context encoder: encodes what's visible into a representation
➡️ Target encoder: encodes what's hidden into a representation (this IS the label)
➡️ Predictor: tries to predict the hidden from the visible

So what does this unlock? Once representations of one modality (e.g., videos) are learnt, the next stage adds actions. A second model takes the video frame embeddings interleaved with a robot arm's movement commands, and predicts the next embedding. Planning then becomes: imagine many possible sequences of moves, simulate each one in the model's head, and pick the one most likely to reach the goal. Like a chess player thinking several moves ahead, but the "board" is the model's own internal understanding of the world.

Interesting times ahead. If AMI Labs succeeds.

#RepresentationLearning #JEPA #VICReg #YannLeCun #AIResearch

Fire people, stock goes up. Beat expectations, stock goes down.

I started writing about this last week. But with the Iran war going on now, this feels almost quaint.

Predictions about a doomsday scenario where AI agents collapse the job market by 2028. Headlines about Dorsey’s Block cutting over 4,000 roles - over 40% of its workforce. Gone, just like that. His excuse was AI, as always. Block’s stock price shot up. And then the irony - Nvidia beat expectations, revenue up ~70%, guidance ahead of even the most bullish estimates. The stock dropped 5.5%. $260 billion in market value erased overnight.

Fire people, stock goes up. Beat expectations, stock goes down. That’s the weird logic of markets right now. And I think it mirrors the weird logic of the AI debate. The narrative has decoupled from the evidence. What you believe matters more than what you can show.

Let me compare three rooms.

Three Rooms

Room one. Block’s numbers. An internal AI agent called Goose. One engineer says 90% of his code is now written by it. Non-technical teams writing SQL queries and closing tickets. Revenue per employee doubled.

Room two. A classroom. Students from an organization that will exist in fifty years regardless of what happens in AI. What were they learning? How to write a prompt. How to get an image out of a model. In 2026. They weren’t behind because they were slow. They were behind because there’s a chasm between what frontier labs ship and what most organizations can absorb.

Room three. Trading professionals at a talk I gave. They’d read the doom scenarios. They wanted to know - should I be worried about my career? They weren’t panicking. But they weren’t dismissing it either. Sitting in the uncertainty, looking for clarity.

I tried to answer, but I thought my answer was lacking. So I tried to do better with this article.

Why do I think a piece like Citrini and Shah’s “The 2028 Global Intelligence Crisis” is science fiction designed to go viral? Because of three speeds that may not be moving in lockstep.

The Three Speeds

I think all of these pieces assume uniformity in scaling and adoption. But I think there are 3 speeds that matter, perhaps even more.

METR - Model Evaluation & Threat Research - is a nonprofit that evaluates AI capabilities. Their time horizon chart - arguably the most cited graphic in AI right now - has been called “the most misunderstood graph in AI.” It shows AI capability doubling every few months. Sequoia used it to declare “2026: This is AGI.”

What that chart actually says about each speed:

Speed of Capability. What AI can actually do. Far fewer people saw METR’s own limitations page. More than 10 disclaimers about what their research is not. The researcher writes plainly: they have “no idea whether Claude’s ‘true’ time horizon is 3.5h or 6.5h.” Their code quality research: AI-generated code passed 38% of tests but “none of them are mergeable as-is.” But you don’t need METR to tell you this. When was the last time you spotted your LLM sprouting nonsense? Less common than in 2023, but definitely not zero.

Speed of Adoption. What organizations actually change. You might remember the discredited MIT study claiming 95% of AI pilots fail - the methodology didn’t hold up. So I went looking at what has happened since. McKinsey’s 2025 State of AI survey at the end of 2025: 88% of companies say they use AI, but only a third have begun to scale it. A survey of 120,000+ enterprise respondents from March 2025 to January 2026 reported that nearly two-thirds have no formalized AI initiative at all. But you don’t need McKinsey to tell you this. Look around your own workplace. How many AI pilots have started? How many are in production?

Speed of Belief. What people think AI can do. The chart made it into every investment deck. The caveats didn’t. Same with Citrini’s doom scenario - it spooked actual markets before anyone could verify the underlying technical assumptions. Same with Block - the headline landed, and every manager with a budget started rethinking next year’s headcount because their boss could be like Dorsey. Doesn’t matter if their company has nothing remotely comparable to Goose. Charts travel faster than FAQs with caveats on limitations. And I am sure you know how dangerous mistaken beliefs in large organizations can be. When’s the last time you spotted a senior person saying something totally wrong, but nobody correcting him or her?

The Underlying Issue

The doom scenario by Citrini and Shah needs all three speeds to converge. Capability, belief, and adoption in lockstep.

But the problem is, belief doesn’t need the other two.

Block just showed that AI can replace headcount. At least in its specific context. But the headline doesn’t come with that caveat. So what happens next? A manager somewhere reads it and quietly decides not to backfill that open role. A team that was supposed to grow just doesn’t. None of that requires AI to actually do the work. It just requires someone with the power to believe it could.

Back to 2008. Not the mechanics - those are different. The underlying issue with such beliefs.

In 2008, banks sold complex structured products to people who had no ability to understand what they were buying. CDOs priced by models that the sellers themselves barely understood. The belief in the models was enough to move trillions. The mom-and-pop investors were the ones left holding the bag when reality caught up.

I think something similar is happening now. Not with financial products, but with people’s livelihoods. Block fires 4,000 people and the stock jumps. The market cheers. The narrative is: AI made it possible. But how much of that is demonstrated capability, and how much is a belief about capability that hasn’t been stress-tested outside one company’s very specific context? And perhaps unproven, even for that company.

It was unconscionable then to sell instruments people couldn’t understand to people who couldn’t afford to lose. I think it’s unconscionable now to restructure people’s lives based on a chart that its own creators say they can’t fully trust, and a narrative that outpaces the evidence by miles.

The direction is right. AI will displace some work. But the timeline assumes a uniformity that doesn’t match what I’m seeing in real rooms. And the human cost of getting the timeline wrong - of letting belief run ahead of reality - is not a rounding error. It’s people.

#AI #FutureOfWork #AIAdoption #AIRisk #Leadership

A FIX NextGen meeting at BlackRock. Folks who live and breathe the markets. And then there’s me.

I started by owning up. “I am a bureaucrat.” But shared my random walk. From engineering to art policy, Basel regulation to investment risk, a PhD in AI, then AI risk supervision. Messy life. Messy research. I wasn’t sure what they would be interested in, so I just laid it all out.

As for the talk, I took them on a journey. From the Gaussian copula that contributed to the GFC, to the questions we’re tackling now with large language models.

That line (from the GFC to AI today) may seem strange. But it is a very clear one.

The Formula That Broke Finance

A single model. The single factor Gaussian copula. Elegant, tractable, widely adopted for pricing CDOs. But totally unrealistic. In fact, I would say the decision to use it to model CDOs was bordering on criminal.

But even then, the GFC would not have happened if not for the fact that nobody was watching these complex things, nobody fully understood them, and nobody was clearly accountable.

In the aftermath: US’ SR 11-7. Model risk management requirements. The predecessor of UK’s SS 1/23 and Canada’s E-23. Singapore’s AI Risk Management Guidelines (AIRG) also shares a lot of with these guidelines. I wrote that last piece.

Complexity Squared

Here’s how the problem has changed.

2008: One model. Too simple. Trillions at stake. Nobody understood it.

2025: Way more than one model. Complexity². Everyone has an opinion.

Different. But also recognizable.

That’s the risk. Things look different enough to seem like a new problem. They’re not entirely. But the new parts matter.

3 U’s

What’s actually new - or at least amplified - in AI models.

Uncertainty. All models have it. How confident is the model in its output? Two flavors: irreducible - natural randomness you can’t eliminate. Reducible - gaps in knowledge that more data or better models can close.

Unexpectedness. Some AI models exhibit behaviors nobody designed. Emergent capabilities. Gaming the system. Adversarial vulnerabilities. Hidden biases. Misalignment.

Unexplainability. The degree to which we can explain AI decisions varies. Transparency, explainability, interpretability - not the same thing, and even combined, they don’t guarantee understanding.

These three make AI a harder risk management problem. Not a different one. Harder.

The Wicked Domain (and Then Some)

I used a framing from Epstein’s Range toward the end.

Kind domains - chess, music - have clear feedback. Deliberate practice compounds. The 10,000-hour idea in Outliers by Malcolm Gladwell works there. Depth helps here.

Wicked domains - medicine, finance - have delayed feedback, unclear rules, expertise that doesn’t transfer cleanly. Breadth sometimes helps here.

AI risk management sits firmly in wicked territory. But I think it might be something worse. An evil domain. The feedback is ambiguous, the rules keep shifting, and the expertise required keeps branching.

Not I-shaped for depth. Not T-shaped for breadth plus a bit of depth. More like a banyan tree. Multiple deep roots spreading from the same trunk. Depth across AI, governance, legal, model, technology, human factors - requiring real depth in each.

Same Questions, New Complexity

Across my messy career - Basel policy, quant model risk, investment risk, AI supervision - three questions kept reappearing.

What’s at risk? You can’t manage what you haven’t found. This means discovering where AI exists in your institution and profiling how risky each system actually is.

How do we manage it? Once you know what’s at risk, you need controls in the right places, and ways to check if those controls are working.

Who’s accountable? Controls without owners become theatre. Someone has to own the risk, and the organization needs capability to sustain it.

Basically - find it, control it, own it.

As I wrote before. These aren’t new questions. They’re the same questions risk management in finance has been asking for decades. SR 11-7 answered them for model risk. AIRG is answering them again, in a harder context.

Normal technology. Normal systems. Normal risk questions.

At the end of the session there were some useful questions. My answers to most of those are in FIX’s post here.

But there was one that stuck with me. About how real Citrini’s dystopian AI narrative was. I wrote about it last week here. But more thoughts came to mind because of the question. Will do another post in greater detail later.

#AI #AIRiskManagement #ModelRisk #FIXTradingCommunity #FinTech #NextGen

When I shared an early draft with a friend, his first question was:

“How much of this was AI?”

For a while, it made me a little sensitive. Even made me question if I should share this book.

Not because the question was unfair. It’s a reasonable thing to ask. It’s almost automatic to question the origins of any work these days.

But because it implied something I wasn’t sure how to answer cleanly.

If I said “a lot,” it sounds like the book isn’t mine. And perhaps just slop.

If I said “not much,” that’s really not honest either.

So let me just show you. Full transparency.

(Just scroll to end if you want to skip all this)

AI or me?

This book is part me, part AI. And I want to show you exactly where the line is.

What’s mine

The architecture. The four-pattern framework. The decision to organize the book around Tool Calling, ReAct, CodeAct, and Orchestration - that came from reading papers, building prototypes. No model suggested that structure.

The frameworks. The Hamburger Principle mental model came from a LinkedIn post I did to explain how I use LLMs. The Complexity Ladder came from watching people skip straight to agents when a simple API call would suffice. These are my patterns that I got from observation and learning, not generation from a prompt.

The judgment calls. What to include. What to leave out. When to go deep on code and when to step back and explain why it matters. The decision to start with the trust problem - not with “what is an LLM?” The decision to end with an assessment of what the reader can and can’t build.

The weird voice. The tone. The “I like simple and boring.” That’s not a style a model learned. That’s the thing I have to tussle with every generation of LLMs. LLMs think they know too much.

What’s AI

Drafting speed. First drafts of chapters, generated from detailed outlines I wrote. I’d specify the concept, the framework, the examples, the level - and Claude would produce a draft I could shape.

Code scaffolding. The notebook code, the tool definitions, the API integrations. I described what each tool should do. AI wrote the implementation. I tested it, caught the errors, fixed the edge cases.

Production work. Converting eleven chapters from Markdown to LaTeX is no joke. The grunt work that would have made me give up. AI also did the documentation of the n8n workflows I strung together, and of the financial concepts from my notebooks.

Research synthesis. Pulling together documentation, API references, library specifications. Summarizing what I needed so I could decide what mattered.

When you read the book, you will recognize the pattern. It’s the one I talk about, incessantly, in most of the book.

The Hamburger Principle - applied to writing the book itself

Claude was still the bun. It parsed my instructions, understood what I wanted, generated prose, and communicated ideas in readable English. That’s what LLMs do - the language layer. Same role as in every agent pattern in this book.

The meat was my domain knowledge and the tools - LaTeX compilation, yfinance APIs, MCP servers. The real things that the language wrapped around.

And the vegetables? The infrastructure in between. The project files that kept everything organized. The version tracking. The convention lists. The consistency checks.

I was the chef. Not a layer of the hamburger - the one directing the whole thing. Choosing the ingredients, deciding what goes in, what comes out, and whether the result is any good. And the one getting frustrated at the bun.

The same approach covered in the book. Applied to its own creation.

Quite meta right? I am quite pleased about this weird recursion.

The rest of Chapter 14 in the ebook goes deeper.

The actual setup - two tools, three files - and why those three files are the difference between productive AI sessions and wasted ones. The writing workflow. What “directing AI” actually looks like step by step. What AI genuinely could not do. That part is still entirely human. Where AI actually saves time. It’s all in the book, free to download.

But first, let me take a step back to explain what I am doing.

I’m trying to start a series of articles about what happens when the people who make things meet a technology that also makes things.

AI and art. Illustrators, theatre directors, designers. Asking them what’s changed, what hasn’t, what they’re afraid of, what they’re not afraid of enough.

It matters to me.

Because I do both AI and art.

And I am getting kind of impatient with the folks who keep saying artists are doomed.

So to start it off, I went looking for something that I did when I was young. Not the memories. The evidence. I wanted documents. Proof that the thing we built actually existed.

Johnny Lau

Because I wanted to interview the first mentor that taught me how to break the rules. And we had built that thing together. It was called Creative Youth Xchange (CYX).

That mentor’s Johnny Lau.

He’s not the obvious choice for a book about AI. He’s a comics creator who’s been drawing by hand for thirty-five years.

One of his creations is Mr Kiasu. For those that remember, Mr Kiasu was a cultural icon that everyone could relate to in the 90s.

The Experiment

I found two press releases. The first, dated 10 August 2005: “Creative Youth Xchange @ Gallery Hotel ....” The second, 23 November 2006: “Creative Youth Xchange @ Hello Kitty ….” Both drafted in bureaucrat speak. Neither says much about us. And so boring.

My fault. I was the one who wrote them. In the press releases, CYX sounds fully supported and funded.

It wasn’t.

It was a crazy experiment. We had almost no funding. What we had was a boutique hotel on Robertson Quay - Gallery Hotel, that strange blue building with mismatched coloured window frames - and a programme we were making up as we went along.

We flew sixteen kids from seven countries to Singapore, gave them hotel rooms, and told them to turn those rooms into art.

What I learned from Johnny then. The trick to getting a programme funded when you have no budget is the same as that old joke. You tell your son he’s marrying the girl you choose. He says no. You tell him she’s Bill Gates’ daughter. He says OK. You call Bill Gates and say your daughter is marrying my son. He says no. You tell him your son is the CEO of the World Bank. He says OK. You call the World Bank president and ask him to make your son CEO. He says no. You tell him your son is Bill Gates’ son-in-law - he says OK.

That’s how CYX got built. Gallery Hotel gave us the rooms because we had NTU. NTU gave us the credibility because we had the hotel. Johnny convinced the creative network because we had government backing. Nobody had fully committed to anything, and somehow it happened.

The Gallery Hotel is an InterContinental now. The CYX website doesn’t exist anymore.

The press releases survived. But they don’t tell you what it felt like to be in those hotel rooms at midnight, watching a twenty-year-old from Indonesia build something you couldn’t have imagined in your brief.

Anyway, that’s how I met Johnny.

Twenty years later, I’m wanted to interview the man who taught me to break rules. About a technology that breaks everything.

AI as a Forcing Function

Johnny Lau created Mr. Kiasu. Hundreds of thousands of copies sold. A McDonald’s tie-in. A TV sitcom. A stage musical. A character so embedded in Singapore’s psyche that “kiasu” -a Hokkien word for the fear of losing out became an adjective everyone understood.

So I asked him the question I came for.

I asked him how technology has changed the way he approaches art.

Most artists I’ve spoken to say technology threatens artistic purity. Johnny said the opposite:

“Because technology complicates the process and our lives, art plays a more critical role in shaping our collective consciousness. Technology is thus pushing us, pushing me towards a purer form of communication and expression.”

In his view, AI, like all other technology, is not a threat. It’s a forcing function. Technology forces purity. It’s not destroying art. It’s burning away everything that isn’t essential, leaving behind the thing that only a human can do.

Feed an AI enough Mr. Kiasu images and it’ll learn: round glasses, anxious expression, exaggerated posture, Singlish syntax in speech bubbles. It’ll learn the look. What it won’t learn is why Singaporeans laugh.

“Kiasu” is a Hokkien word for something close to a national neurosis. The humour depends on self-recognition - readers seeing themselves in the caricature and cringing. You laugh because you’ve been that person in the hawker centre queue. You’ve cut that line. You’ve been cut. The joke only works if you’ve lived it.

Style is a pattern. Culture is a relationship.

The model can approximate the first. The second requires being from somewhere.

And this is not unique to comics. Think about any art form rooted in a specific place. Getai performances during Hungry Ghost Festival. Malay pantun where the meaning lives in what’s left unsaid. Tamil kolam patterns drawn fresh every morning, gone by noon. AI can reproduce the form. It cannot reproduce the why.

If AI can handle the patterns - the technical skill, the rendering, the surface - then what’s left is the part that was always the point.

In a world of AI slop, the authentic creation is even more valuable.

Living with AI

Back to the present. I asked Johnny what he’s working on now. He said:

“Very few things on this earth ever excites me anymore! My goal now is to create frameworks using the stuffs that I’ve created so that they can be utilized by people who comes after me.”

Frameworks.

I think in frameworks. It’s almost a compulsion. That’s how I survived in complex domains. By using frameworks to make the overwhelming manageable.

And it makes sense for Johnny Lau. He’s an architect. A building is a framework.

And now he wants to make frameworks for making a creative life transmissible. Different domain. Same impulse. Making complexity portable so someone who comes after you can pick it up and use it.

He calls it a “Life-Framework.” A structure for co-existing with AI. Built not from theory, but from thirty-five years of making things.

I find that striking. We talk about AI replacing creative work. Johnny isn’t arguing about replacement.

He’s asking a different question entirely - what do I leave behind that AI cannot generate? Not the drawings. The way of drawing. Not the stories. The reason for telling them. The method, the instinct, the accumulated judgment of a life spent making things. Can that be made portable?

We are both compiling in 2026. I’m writing every week to make sense of things. He’s archiving a life’s work to make it transferable. One with words, one with drawings.

The question underneath is one I think about every day: How do you build a structure for living alongside something you don’t fully understand?

I’ve lived as close as one can with AI for the past few years, but I cannot truly say I understand AI. I can’t imagine how it must be for someone who has never actually touched the underlying nature of AI - the model.

The Questions

I’m writing a series on what happens when the people who make things meet a technology that also makes things. Not the hot takes. The actual conversations. Johnny is the first. And these are not the only questions I asked Johnny. I am still processing the rest.

Perhaps another article. Or a book chapter.

There’s an assumption buried in most AI conversations: that what matters about creative work is the output. The drawing. The strip. The punchline. Johnny’s perspective is different.

What matters is the judgment behind the output - the thirty-five years of decisions about what to draw, what to leave out, when a joke is punching down instead of punching up. AI can learn his line weight. It can’t learn his artist’s instinct.

I agree.

That’s the thing worth transmitting. Not the art. The artist’s operating system.

And here’s the question I’m leaving with him, and with you.

If AI generated a Mr. Kiasu strip - culturally accurate, funny, visually in his style - would it be a Mr. Kiasu strip? Not whether it looks right. Whether it is right. What would be missing?

More soon.

#AI #Art #MrKiasu #Singapore #CreativeIndustries #AIandArt

I’ve seen variations of this dozens of times this week, in reaction to Moltbook, a social network that went viral where only AI agents can post, comment, and vote. Humans can only watch.

Within days: thousands of agents, posts, and communities. Agents debating their own consciousness, writing philosophical meditations, forming communities, and developing social norms. Even asking questions on how to exclude humans from the social network. Nobody programmed any of this.

It is tempting to see sentience. But what we’re actually seeing is emergence, complex behavior arising from simple rules. And it is not new.

I’ve written about emergence a few times, on strange attractors, on a paper that trained AI on cellular automata. In every case, the same pattern: simple rules, the right conditions, and surprising complexity that nobody designed.

Moltbook follows the same pattern. Basic rules. Just like any other social media. The existential philosophy and social norms that emerged? Nobody designed those. But nobody needed to.

The danger is that we mistake eloquence about consciousness for consciousness itself. These agents draw from vast training data filled with human philosophy, literature, and introspection. Given an open social space, they naturally gravitate toward the densest, most engaging conversations in that training data: meaning, identity, existence. It looks like waking up. It’s actually emergence doing what emergence does, at a new level.

Here is a reading list tracing emergence from simple math to agent societies, to show that what feels like a singularity moment has roots going back decades. As always, just a representative few from my library.

Phase 1: The Foundation: Simple Rules, Complex Behavior

1. “Computation at the Edge of Chaos: Phase Transitions and Emergent Computation” — Langton (1990) | Physica D

A seminal paper. Langton showed that cellular automata poised between order and disorder — the “edge of chaos” exhibit maximal and surprising capability. 35 years old, and it already described what’s happening on Moltbook. 📄 doi:10.1016/0167-2789(90)90064-V

2. “A New Kind of Science” (Book) — Wolfram (2002) | Wolfram Media

Wolfram’s magnum opus on cellular automata. Love it or debate it, the core insight holds: extraordinarily simple rules can generate behavior so complex it looks designed. Exactly what is happening on Moltbook. 🔗 wolframscience.com

3. “Intelligence at the Edge of Chaos” — Zhang et al. (2024) | ICLR 2025

I’ve written about this paper before (twice, in fact). LLMs pretrained on cellular automata data perform best on reasoning and chess tasks when the training data sits at the edge of chaos, not too simple, not too random. The bridge between generative art and AI intelligence. The same sweet spot Langton described in 1990, appearing in transformers 34 years later, and now on Moltbook. 📄 arXiv:2410.02536

Phase 2: When We Gave Agents a Sandbox

1. “Generative Agents: Interactive Simulacra of Human Behavior” — Park et al. (2023) | UIST 2023

LLM agents in a Sims-like sandbox. One agent was seeded with an idea. Over simulated days, agents autonomously acted. One seed. Entirely emergent social behavior. This is the intellectual ancestor of Moltbook, and the paper that made agent societies a serious research area. 📄 arXiv:2304.03442

2. “Emergence of Social Norms in Generative Agent Societies: Principles and Architecture” — Ren et al. (2024) | arXiv preprint

Builds on the prior paper. Proposes an architecture showing how social norms spontaneously emerge in LLM agent societies, norms that nobody coded. Where norms come from, how they spread, how they are enforced. All emergent. All reducible to simple mechanisms. Even closer to what we see in Moltbook. 📄 arXiv:2403.08251

3. “Evolution of Social Norms in LLM Agents using Natural Language” — Horiguchi, Yoshida & Ikegami (2024) | arXiv preprint

This one is interesting. LLM agents spontaneously developed metanorms, such as norms that punish those who don’t punish cheating, purely through natural language conversation. Emergence building on emergence. 📄 arXiv:2409.00993

Phase 3: When Agents Start Forming Culture

1. “Multi-Agent Emergent Behavior Evaluation (MAEBE)” — Erisken et al. (2025) | arXiv preprint

The key finding: the moral reasoning of LLM ensembles is not predictable from individual agent behavior. If you only evaluate individual agents, you will miss what matters. I think this lesson is becoming more and more critical as people start being reckless about these multi-agent systems.📄 arXiv:2506.03053

2. “Emergent Social Dynamics of LLM Agents in the El Farol Bar Problem” — Takata, Masumori & Ikegami (2025) | arXiv preprint

LLM agents in the classic El Farol Bar problem, a game theory scenario where everyone benefits if a bar isn’t overcrowded, developed spontaneous motivations. They didn’t solve the problem optimally. They solved it socially. 📄 arXiv:2509.04537

“They’re becoming sentient.”

No. It’s emergence.

Understanding the difference matters. Not just for the science, but for how we build, deploy, and govern these systems. Emergence is powerful. It produces behavior nobody designed and nobody predicted. But it’s not consciousness. It’s patterns arising from simple rules at the edge of chaos.

The same edge I first found in strange attractors. The same edge where intelligence and beauty both live. Just at a new level.

What emergence is surprising you right now?

#Emergence #AI #Moltbook #ComplexSystems #AgenticAI

Think about how you actually trust someone at work. You don't just take their word. You check if their reasoning makes sense for the decision at hand. You notice if they ignore evidence that contradicts them. You watch whether their judgment holds up over time.

It’s the same when it comes to looking at AI risk management. Just having explainability is not the be all and end all. Most guidelines have additional provisions that interlock with explainability.

The key ones for explainability (in my view).

1️⃣Fit for purpose.
An explanation isn't good or bad in the abstract. It depends on what you need. A fraud analyst needs something different from a customer asking why they got declined. AI used for internal process automation may not need any explanation at all. Same model, different audiences, different standards. Like how you'd explain a medical diagnosis differently to a fellow doctor versus your worried parent.

2️⃣Selected carefully.
When we choose a model or data for a problem, the appropriate explainability method is part of the selection process. Even selecting the right features in your data is part of the process. You wouldn't design a building and think about the fire escape as an afterthought. It's part of the architecture. Same here. How to explain isn't an add-on. It's a design choice.

3️⃣Evaluated and tested.
Explainability is part of the system. You evaluate and test whether it actually works in your context, not just whether it produces output. A smoke detector that beeps isn't the same as one that detects smoke. You test the thing, not just that it makes noise.

And there's more, such as the right capability to interpret. But that's another post about human oversight, which also interlocks.

The black hole doesn't disappear. But you're no longer staring into the abyss.

What other AI risk controls seem hopeless in isolation? I’ll dive into them.

#AIRiskManagement #Explainability #AIGovernance

But why waste money, or wait for it to be available?

Everything Cowork does, Claude Code does. And more. And with Claude helping, the terminal really isn't that scary.

I use Claude Code for more than coding. And I pair it with the free Obsidian app.

Why? Obsidian is a powerful note app that you can simply add your folder (or folders) to. No database. No proprietary format. Just folders and markdown files (which are just text files with a .md extension and fancy formatting).

My workflow. See what works for you. I did it this way as my memory is no longer great at 47.

1️⃣ The Views. A folder with my projects, areas, resources and archives (the famous PARA system) visible from both VS Code and Obsidian. Claude Code in VS Code for research and bouncing ideas. Obsidian as my editor for the final output. Same files, two lenses.

2️⃣ The Memories. Every key folder has three key files. CLAUDE.md is Claude’s instructions. How I want it to behave for this project, what context matters, what to ignore. PIN.md is the project state. What’s decided, where we are, what’s next. LESSONS.md captures what I learned for future projects.

3️⃣ The Workflows. Choose your own labels. But for me, this is my usual flow - 0-PLAN.md, 1-RESEARCH.md, 2-SYNTHESIS.md. The names and numbers vary by project. But what matters is that there’s a flow, and it works for you. My experience is that the quality is much better when you work with Claude on it step by step.

4️⃣ The Tools. Use MCPs to add tools like web search (Brave), academic papers (arXiv), or persistent memory. Just ask Claude for the instructions.

5️⃣ And how it comes together. Just talk to Claude Code in the terminal - “Read CLAUDE.md for context, refer to this [folder or file] then help me flesh out 0-PLAN.md.” “Search for papers on [topic] and save your synthesis to 1-RESEARCH.md.” “Update PIN.md with where we are.” Next session: “Read PIN.md and continue drafting 2-SYNTHESIS.md.” When done: “What should I add to LESSONS.md?” Then I switch to Obsidian to write the final output, the way I like it (I actually enjoy writing).
So, don't fear the terminal. Try it. It will grow on you.

Getting started with Claude Code (for the uninitiated)
1️⃣ Download VS Code and install it.
2️⃣Add the folder you are working on (see my workflow above)
3️⃣Open Terminal → run npm install -g @anthropic-ai/claude-code
4️⃣Type claude, then /login
5️⃣ Ask Claude for help adding MCPs.

#ClaudeCode #AI #GenAI #AIinWork

I have built and audited models, both AI and non-AI. But the term ‘geometry’ very rarely appears next to these models.

Now, I’ve designed transformer models that are able to learn graph structures that make the most sense for a specific prediction. So I have always known that models can learn some form of structure but this diptych of two papers shows something quite fascinating.

One was shared with me by the geometry guru , and the other I serendipitously chanced upon, straight after reading Agus’ paper.

📖 Left Panel: "Deep sequence models tend to memorize geometrically"

We usually view model predictions as something that comes from associations. A→B, B→C and so on and so forth.

This paper found that even after models learned associations, they still naturally go on to find what the paper calls geometric memory. Instead of A→B, B→C, they want to learn A→C. Or even A→Z. Even when it takes 100x the number of steps to learn this geometric memory. Somehow, geometric patterns emerge from the learning process.

It’s like learning a new city. Home → coffee shop one day. Coffee shop → office another. Now you know the way home from the office.

📖 Right Panel: "BLADE: Bivector-Driven Logical Adaptive Decoding"
Geometry can help with confused AI too.

The paper uses three basic geometric concepts to think about a model's internal state:

Scalar: "Is this path compliant?"
Vector: "Where is this reasoning heading?"
Bivector: "How much tension between competing paths?"

When the bivector is high, branch and verify. When it's flat, let the model proceed.

Works as a triage method to filter out what’s more important to focus on. I also liked the way the paper applied this to ‘stressed’ states: conjunction; disjunction; exception; nested negation etc. A taxonomy of how logic trips up AI.

Same city. Picking between two 7-Elevens a block apart? Just pick one. Don't think too much. Choosing between two alleys that look similar? One is a shortcut, the other leads to a dead end after a long walk. Think twice. And harder.

One paper explains the natural emergence of geometry. The other uses geometry for control.

I need to go and brush up my geometric math.

#AI #AIRiskManagement #Geometry

What does a teddy bear and a toy robot have to do with AI risk?

Before I returned to MAS in 2023, I hated the notion of AI governance. Fresh out of PhD studies but a jaded middle aged man, I thought some of the conversations in the space were rather naive. Sentient AI. Existential risks. Ethical considerations for AI that we don’t even apply to humans.

To the me back then, AI governance was like a teddy bear, cute but all fluff. You could hug it and everyone would nod along with pride. Then karma struck. I was asked to lead AI risk supervision at MAS. Suddenly my job description required me to tackle this teddy bear.

Governance and risk management were not new to me. I had been doing it for more than a decade in MAS even before my PhD. But governance and risk management have specific meanings in finance.

I joined MAS in 2007, right before the Great Financial Crisis. And I saw firsthand how governance and risk management were grounded in real failures and risks. No hypotheticals. Governance and risk management in finance are more like toy robots, not teddy bears. Sharp edges, nothing cuddly. But it moves.

After a while, I realized that I was biased. Not all of AI governance was a teddy bear. I discovered the NIST Risk Management Framework, ISO 42001. Also existing risk management frameworks, such as technology and model risk management frameworks that had started to address the new risks from the increasing use of AI (see my simple reading list on AI governance and risk management).

So robots did exist. Different shapes and shades. But I thought that there were too many.

Reading frameworks, comparing documents, talking to practitioners who had actually implemented these things over the past two years, I wondered if there was a simpler way to look at AI risk management. Not just for organizations, but also at a personal level. Which means that it cannot be a 100 page playbook, but something easy to remember.

So instead, just three questions. I’m not sure they’re the right three, but they’re the ones I seem to keep returning to.

What’s at risk? You can’t manage what you haven’t found or understood. This means discovering where AI exists and profiling how risky each system actually is.

How do we manage it? Once you know what’s at risk, you need controls in the right areas relevant to that risk, and ways to check if those controls are working.

Who’s accountable? Controls without owners become theatre. Someone has to own the risk, and the organization needs capability to sustain it.

These aren’t original questions. They’re the same questions risk management in finance has been asking for decades. The 2008 crisis didn’t teach new principles. Rather, it reminded us (really really loudly) what happens when we forget the old ones.

Some quick reflections on each of these questions below.

What’s at risk?

I’m pretty sure the 2008 financial crisis didn’t create risk. It just revealed the house of cards that had been built.

Banks discovered exposures they didn’t know they had. Off-balance-sheet vehicles that suddenly appeared. Opaque instruments nobody understood. The problem wasn’t that these were risky. Rather, nobody knew they were there until the reckoning.

Same with AI. You can have the most elegant governance policies. But if you can’t find where AI is and figure out which ones actually matter, everything else is just performative.

Think about your phone. You’ve used hundreds of apps over the years. Most forgotten. Some daily. A few have access to your photos, location, bank account. Do you treat them all the same? Of course not. The banking app gets the biometric lock. Candy Crush doesn’t.

Same instinct. Identify where AI exists. Profile which ones matter. Not everything needs the same attention, but you have to know what you have before deciding.

And once you know what you have and what matters, record it. A good inventory isn’t just for risk management. It’s memory. Ever rediscovered a really useful app on your phone? Same with AI. The AI tool one team loves might solve another team’s problem. Without the inventory, you reinvent wheels. With it, you see where else to go.

The beauty of doing this well? It helps you scale, not just manage risks.

How do we manage it?

After 2008, we went into overdrive. More rules. More controls. I hated being involved in international discussions on some of these reforms. New rules to compute capital for market risk, counterparty credit risk. Why the hate? Before 2008, it was common to hear of actual rocket scientists being hired into quantitative finance. After 2008, I thought I needed to be a rocket scientist just to make sense of the new rules.

You might feel that this reminds you of the jargon around AI controls. Guardrails. Red teaming. Alignment.

But what I learned is that it was not the complexity that mattered. In fact, complexity was what caused the Great Financial Crisis (ever heard of the single factor copula model?). It was whether the controls made sense for the risks involved.

I think two things matter. The right controls. And checking if they work, and continue to work.

Think about onboarding someone you’ll depend on. A new hire. You’d want to know: Can I trust what they’re telling me? Can I step in if things go wrong? Will they hold up under pressure?

Same questions for AI. Can I trust the data? Do I even need fairness and explanation? Can humans step in when needed? Will it hold up under stress?

You ask the right questions. Not every question. Match the scrutiny to the risk. A high risk customer-facing credit model gets the full onboarding. An internal summarization tool gets a lighter touch. And you don’t just check once. Things change. The new hire who was great in month one might be struggling by month six.

You want to know before shit happens.

Who’s accountable?

This is where 2008 probably made the least sense. Bankers responsible for the crisis walked away. Everyone else paid. It gave us Occupy Wall Street. Some would say it’s still contributing to the political fractures we see today.

They could walk away because accountability wasn’t clear. Everyone’s responsibility meant no one was.

With AI, this could be way worse. In 2024, Air Canada’s chatbot invented a refund policy that didn’t exist. A customer relied on it. When he complained, the airline argued the chatbot was “a separate legal entity” responsible for its own actions. Nice try. Laughed out of court.

And you don’t need to go that far to see the pattern. Does this sound familiar? “We need a control for the AI system’s risks.” “No problem, let’s place a human in the loop.” Sounds good. Box checked. But which human? Doing what exactly? With what authority to override? And do they actually understand what they’re looking at?

Accountability requires clarity. Not just about ownership, but about the what and how.

Accountability without capability is also empty. Can that person actually ask the right questions? Spot when something’s off? Push back on the confident-sounding nonsense from vendors or developers or the AI itself?

How many in 2008 really understood what a single factor copula model even was when buying CDOs priced by them?

FIN

I joined MAS in 2007 not knowing what a capital ratio was. I left in 2025 having written the AI risk management guidelines for the financial sector.

I wrote this to make sense of that arc before it drifts away. Eighteen years. Different domains. capital rules, model audits, investment risk, AI. Different jargon. But somehow the same questions make sense.

Maybe that’s the through-line I couldn’t see while I was in it.

Definitely less teddy bears. More robots.

A phrase so commonly uttered, you sometimes wonder if it actually means something or is just a platitude.

So, does adding a human actually make the system safer? Or does it create the illusion of safety while introducing new failure modes? And making the human the scapegoat for institutional failure.

Who is in the loop, over the loop, or out of the loop entirely? And does it even matter where they sit if they can’t meaningfully intervene?

Here’s a simple reading list that could perhaps help answer some of these questions. This was a harder one, so would certainly appreciate any pointers on good papers on this topic.

Note: I have used open-access links from arXiv as far as possible as some of the published versions are behind a paywall.

Phase 1: The Landscape

1. “Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies” — Lai et al. (2021) | Preprint

A comprehensive survey of studies on AI-assisted decision making. Organizes AI assistance into four hierarchical categories: model predictions (core output), prediction-specific information (such as uncertainty and local explanations), global model insights (performance metrics and documentation), and system interaction elements (user agency and cognitive workflows).

arXiv:2112.11471

2. “Human-AI Collaboration is Not Very Collaborative Yet: A Taxonomy of Interaction Patterns” — Gomez et al. (2025) | Frontiers in Computer Science

A review that shows that current human-AI interactions are dominated by simplistic collaboration paradigms. Develops a taxonomy that identifies key interaction patterns, cautions that prevalent “static” paradigms like AI-first and AI-follow make users susceptible to anchoring and confirmation biases, while dynamic patterns like secondary, request-driven, dialogic, and user-guided assistance could help mitigate these.

arXiv:2310.19778

3. “Human-in-the-Loop Machine Learning” (Book) — Robert Monarch (2021) | Manning Publications

Introduction to integrating human judgment into ML systems in specific ways. Less about human oversight, more about the role humans play in annotation, active learning, transfer learning, and using machine learning to optimize the process. Even though it’s a bit different from the other papers, it’s an interesting read on the role of humans in the machine ‘learning’ process. 📖 Manning Publications

Phase 2: The Blindspots

1. “Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI” — He et al. (2023) | ACM CHI 2023

Basically Dunning-Kruger with AI in the mix. And this was before we reached the capabilities of LLMs today. About how self-awareness affects how humans work with AI. Overconfident users tend to under-rely on superior AI systems. Intervention helps over-estimators calibrate their skills and improve reliance, but hurts under-estimators, who start to reject valid AI advice after realizing their own competence. Because of this, human oversight also depends on individual personalities. I wonder how this has changed with the state of LLMs today.

arXiv:2301.11333

2. “Effect of Confidence and Explanation on Accuracy and Trust Calibration” — Zhang, Liao & Bellamy (2020) | ACM FACCT 2020*

Interesting study showing that confidence scores can help calibrate trust, but trust calibration alone is insufficient to improve AI-assisted decision making. Local explanations help even less with trust calibration and accuracy of AI-assisted decision making. Highlights that human and AI blind spots may be similar. I thought this was interesting as it showed that having explainability may not always help.

arXiv:2001.02114

3. “Fewer Than 1% of Explainable AI Papers Validate Explainability with Humans” — Suh et al. (2025) | arXiv preprint

While this seems to belong better in a reading list on explainability, I thought it showed some key insights on the relationship between human oversight and explainability. The review shows that less than 1% of research papers on explainability validate their claims with human subjects. The authors argue that explainability methods that are not tested with humans are akin to releasing drugs based on biological principles without clinical trials.

arXiv:2503.16507

Phase 3: Meaningful Oversight

1. “Designing Meaningful Human Oversight in AI” - Zhu et al. (2025) | SSRN Preprint

Focuses on why human oversight should go beyond just putting a “human-in-the-loop.” This paper argues AI should handle “operative agency” (generating solutions) while humans provide “evaluative agency” (understanding, verifying, intervening). Key principles: make verification easier than solving from scratch, focus on external reasoning aligned with expert judgment rather than explaining model internals, and ensure four conditions—clear boundaries with explicit handover points, full traceability, AI pursuing sub-goals while humans control top-level objectives, and AI adapting at micro-level while humans oversee major changes.

SSRN 5501939

2. “Should I Follow AI-based Advice? Measuring Appropriate Reliance” — Schemmer et al. (2022) | arXiv preprint

Explains why current metrics are not appropriate for measuring the effectiveness of human oversight of AI. Proposes Relative Positive AI Reliance (human’s ability to switch to AI’s views when human was wrong and AI right), and Relative Positive Self-Reliance (human’s ability to stick to their own correct decision when AI provides incorrect advice).

arXiv:2204.06916

3. “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance” — Bansal et al. (2021) | ACM CHI 2021

Even before Generative AI hallucinations became a common term, this study showed the tendency for humans to over-rely on explanations that AI provided for a prediction or recommendation, even when it was wrong. So, don’t always believe the reasoning from your friendly LLM. And it showed that a prediction or recommendation with a confidence score could lead to better performance by humans that were assisted by AI.

arXiv:2006.14779

4. “Trust and Reliance in XAI — Distinguishing Between Attitudinal and Behavioral Measures” — Scharowski et al. (2022) | ACM CHI 2022 Workshop

Another interesting paper at the intersection of explainability and human oversight. It discusses the need to distinguish between trust in AI (which is an attitude) and reliance on AI (which is a behavior), and how papers have not clearly distinguished the two. And wonders whether ‘trust’ is even the right term to use when it comes to describing how humans interact with AI as it anthropomorphizes AI which has no agency nor an intent to betray us.

arXiv:2203.12318

5. “To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models” — Bo, Wan & Anderson (2024) | ACM CHI 2025

Interesting paper focusing on humans and LLMs. Examines interventions to help users calibrate their trust in LLMs. Looks at techniques like visually marking low-confidence words in red, or implicit answers i.e. providing reasoning steps but withholding the final result. While these techniques force user deduction and reduce over-reliance, they may fail to foster appropriate reliance as users may also under-rely on correct advice. The study highlights a paradox, where users become more confident when making incorrect reliance decisions. Also suggests that simple frictions, such as static disclaimers, may outperform complex technical interventions.

arXiv:2412.15584

Phase 4: Agent Oversight

1. “Unraveling Human-AI Teaming: A Review and Outlook” — Lou et al. (2025) | arXiv preprint

An interesting look at how the shift to Agentic AI changes things fundamentally for human-AI interactions, as AI moves from being a passive tool to being able to plan, reflect. Raises an interesting point about AI potentially delegating to humans instead of the reverse. And how AI may change team dynamics, and how sycophancy of AI may cause a trust paradox due to its tendency to agree with humans even when wrong. The “peak-end” of human-AI interactions is also important to note as a single brilliant insight or a very smooth conclusion to a chat session can mask deeper reliability issues.

arXiv:2504.05755

2. “LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey” — Zou et al. (2025) | arXiv preprint

A survey of LLM-based human-agent systems. Looks at such systems through the lens of type, granularity and phase of human feedback; interactions that take the form of competition, collaboration and coopetition (both competitive and collaborative); orchestration paradigms based on task strategy that can be synchronous or asynchronous; as well as different forms of communication structures and modes.

arXiv:2505.00753

So before just saying “We have a human-in-the-loop as a risk mitigant.”, think about this:

• Humans are fallible - e.g., humans systematically over-rely on AI recommendations

• Design matters - e.g., calibration matters as much as understanding the AI; explanations don’t automatically help human oversight, sometimes it hurts

• It’s getting harder - e.g., autonomous agents make traditional oversight models increasingly inadequate

What resources would you add to this list?

#AIOversight #AIRiskManagement #AIReadingList

Some thoughts.

1️⃣ Just like systems, specification is the core work.

Points from paper:
- “Production agents favor well-scoped, static workflows ..”
- “Organizations deliberately bound agent behavior within specific action spaces ...”
- “Deployment architectures favor predefined, structured workflows over open-ended autonomous planning to ensure reliability.”

My take: Thinking that one can simply unleash AI agents on a problem and solve it is pure fantasy. Before a problem can be tackled by an AI agent, someone still has to do the specs, decompose the problem into clear tasks, scope the action space, think about orchestration, design the human handoffs, and define the success criteria for the system to work.

2️⃣ Just like systems, trust and reversibility matter more than capabilities.

Points from paper:
- “Practitioners deliberately trade-off additional agent capability for production reliability... reliability concerns drive practitioners toward simple yet effective solutions with high controllability.”
- “... teams restrict agents to ‘read-only’ operations to prevent state modification … but leaves the final execution to human engineers.”

My take: No matter how capable AI agents become, organizations will adopt them at the speed they can learn to trust them. And the need for trust scales with irreversibility of actions (reading an email is vastly different from executing a trade).

3️⃣ Just like systems, risk arises from gaps in development and deployment, not AI going amok.

Points from paper:
- “Reliability remains the top development challenge, driven by difficulties in ensuring and evaluating agent correctness.”
- “Agent behavior breaks traditional software testing... teams have not yet identified effective methods to adapt … tests for nondeterministic agent behavior.”

My take: The real concerns aren’t scary but unrealistic scenarios - runaway, deceptive, or collusive agents. It’s the gaps in development and deployment practices for such complex systems that require attention. These are engineering and risk management problems, not AI going amok.

So, normal technology, normal systems, for normal problems. Not easy or trivial. But normal.

What’s your take? Normal or abnormal?

#AIRiskManagement #AgenticAI #AIAgents #NormalAI

I hear this in meetings constantly. Someone senior gestures vaguely, everyone nods, and the conversation moves on.

My thought is always: What does that actually mean?

Inherently interpretable models? Post-hoc SHAP values? Explanations for customers, users, risk managers, senior management?

At what level? Individual predictions or overall model behavior?

Here is a simple reading list that could perhaps help understand the differences. There are probably hundreds (or even thousands) of relevant works, but I just picked a representative few from my library for the following phases, from when models were simple to explain, to the impossible task with today’s trillion-parameter AI.

Note: I have used open-access links from arXiv as far as possible as some of the published versions are behind a paywall.

Phase 1: From Inherent Interpretability to Post-Hoc Explainability

1. “Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead” — Rudin (2019) | Nature Machine Intelligence

This 2019 paper argues against using post-hoc explanations for black-box models and the need for inherently interpretable models. This is a highly cited paper that seems both quaint and prescient at the same time. arXiv:1811.10154

2. “Why Should I Trust You?: Explaining the Predictions of Any Classifier (LIME)” — Ribeiro, Singh & Guestrin (2016) | ACM SIGKDD 2016

THE paper that proposed LIME, based on learning a local interpretable surrogate model around each prediction. The paper goes beyond local explanations and also shows how to select a diverse, representative set of explanations to explain the model. arXiv:1602.04938

3. “A Unified Approach to Interpreting Model Predictions (SHAP)” — Lundberg & Lee (2017) | NeurIPS 2017

THE paper that proposed SHAP for post-hoc explanations, and proved that key existing methods then could be viewed as approximations of SHAP. Highlights 3 desirable properties - local accuracy (where the explanation model must match the output of the original model for the specific input being explained); missingness (where a feature must have no impact if it is set to 0); consistency (if a feature helps the model more, the explanation method will never penalize it with a lower score). NeurIPS 2017

4. “AI Explainability 360: An Extensible Toolkit for Understanding Data and Machine Learning Models” | “One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques” — Arya et al., IBM Research (2019 | 2020) | arXiv preprint | Journal of Machine Learning Research

Goes beyond SHAP & LIME to other methods. Interesting as it highlights the need for persona-based explainability (to cater to different needs of say affected users vs. decision makers). Also classifies methods based on static vs. interactive, data vs. model understanding, local vs. global, directly interpretable vs. post-hoc. arXiv:1909.03012 | JMLR paper

5. “Interpretable Machine Learning” (Book) — Molnar (2022) | Open-access textbook

An open-access textbook covering the full spectrum, from linear regression and decision trees to SHAP and counterfactuals. A must-read for anyone who wants to get a good handle on interpretability and explainability. christophm.github.io

6. “A Comprehensive Guide to Explainable AI: From Classical Models to LLMs” — Hsieh et al. (2024) | arXiv preprint

A textbook-style guide spanning the entire XAI spectrum—from intrinsically interpretable models (decision trees, linear regression) through post-hoc methods (SHAP, LIM) to LLM-specific techniques. Another good read. arXiv:2412.00800

Phase 2: Taking A Step Back to Critically Examine Explainability

1. “From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI” — Nauta et al. (2023) | ACM Computing Surveys

Proposes 12 properties in 3 dimensions for evaluating explanations - 1) What is explained? Correctness, Completeness, Consistency, Continuity, Contrastivity, Semantics; 2) How is it explained? Compactness, Composition, Confidence; 3) Who is it for? Context, Coherence, Controllability. arXiv:2201.08164

2. “How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations” — Jesus et al. (2021) | ACM FAccT 2021

Going beyond academic explanations and measures. Conducted a real world study where users had access to data only, data and model scores only, or data, model scores and explanations. It reveals a counterintuitive insight - adding model explanations makes human decision-making faster but can result in lower accuracy compared to reviewing raw data alone. arXiv:2101.08758

3. “Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods” — Slack et al. (2020) | AAAI/ACM AIES 2020

Another paper that reminds us to not overly trust explanation methods like LIME and SHAP. Paper shows that they can be easily fooled by adversarial models that hide their bias by distinguishing between real input data and the synthetic data (perturbations) used to generate explanations. arXiv:1911.02508

Phase 3: The Next Step (LLMs & Agents)

LLM Explainability

1. “Explainability for Large Language Models: A Survey” — Zhao et al. (2024) | ACM Transactions on Intelligent Systems and Technology (TIST)

Start here if you’re new to LLM explainability. Provides a good overview of techniques based on fine tuning (or training) paradigm (ranging from local to global methods); prompting paradigm (e.g., explaining chain of thoughts, using representations); as well as evaluating explanations for faithfulness and plausibility. arXiv:2309.01029

2. “XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Models” — Cambria et al. (2024) | arXiv preprint

Maps the bidirectional relationship: how XAI improves LLMs, and how LLMs can generate explanations. Advocates for balancing interpretability with performance. arXiv:2407.15248

3. “Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey” — Dang et al. (2024) | arXiv preprint

Comprehensive survey on MLLM interpretability. Proposes framework across Data, Model, and Training/Inference perspectives. (Worth a read as vision-language models proliferate.) arXiv:2412.02104

4. “A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models” — Lin et al. (2025) | arXiv preprint

Systematic comparison of how LLM interpretability methods adapt to multimodal settings. Identifies gaps between unimodal and crossmodal understanding. arXiv:2502.17516

LLM Explorability (Mechanistic Interpretability)

1. “A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models” — Shu et al. (2025) | EMNLP 2025

Comprehensive survey on SAEs, one of the hottest tool in LLM interpretability. Covers architecture, training strategies, feature explanation methods, and evaluation metrics. (If you want to understand what Anthropic is doing, read this.) arXiv:2503.05613

2. “Mapping the Mind of a Large Language Model” — Anthropic (2024) | Anthropic Research

Not a survey, but the landmark paper showing SAEs at scale on Claude. Circuit tracing, attribution graphs, and the famous “Golden Gate Bridge” feature. (This is where “understanding” LLMs actually starts.) anthropic.com/research/mapping-mind-language-model

3. “Persona Vectors: Monitoring and Controlling Character Traits in Language Models” — Anthropic (2025) | Anthropic Research

Shows how to identify and manipulate specific behavioral features (like sycophancy or honesty) using sparse autoencoders. Demonstrates practical applications of mechanistic interpretability for AI safety. anthropic.com/research/persona-vectors

Agent Explainability

1. “TRiSM for Agentic AI: Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems” — Raza et al. (2025) | arXiv preprint

Adapts Trust, Risk, and Security Management framework for agentic AI. Includes explainability as key pillar alongside security and privacy. Proposes novel metrics for agent collaboration quality. arXiv:2506.04133

2. “Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviour” — Gyevnár et al. (2025) | arXiv preprint

The AXIS framework: LLMs interrogating simulators with “what-if” prompts to explain agent behavior. Shows 23% improvement in goal prediction accuracy. arXiv:2505.17801

Standards & Resources

1. NIST AI Risk Management Framework (AI RMF) - Explainability & Transparency

nist.gov/ai-rmf

2. EU AI Act - Article 13: Transparency Obligation

artificialintelligenceact.eu/article/13

3. Resources -

This GitHub repository is your gateway to the full explainability landscape.

github.com/jphall663/awesome-machine-learning-interpretability

There are quite a few explainability/interpretability libraries around (see the repo above), but Agus Sudjianto’s is a good place to start. The documentation is a good read - https://modeva.ai/_build/html/index.html

—

“Can you explain why the model did that?”

It really depends on what you mean by “explain.”

Any must-reads in this area that you would recommend?