I went back to my old workplace to teach this past week. After nine months out of a place I spent 18 years in.
After a full day of training, I found myself thinking about the habits I’d picked up there.
A full week otherwise. Two full days of training. One for my old workplace. One for the Association of Banks in Singapore. A short book draft finished and handed to my collaborator. A few engagements locked in for the coming months. A few no’s, some received and some given. A reporter asking for views on an AI vendor. Decks late into the night, and a watercolor before bed when I could. And lots to read about rogue AI agents and some weird thing called pacing. An old friend based in New York even called me one night to ask for my take.
Much of what I learned doing policy work still helps me make sense of the world. Some reflections on those habits, and how they apply to AI and independent life.
Boring habits from policy work
I’ve worked on policy work in arts, banking regulation, board corporate governance, financial risk management, and AI risk management. Along the way, I drafted acts, regulations, notices and guidelines, did inspection reports, and wrote board papers for people who had to make decisions. Different subjects, but much of what I learned was transferable.
Think from first principles
In policy work, it’s tempting to start with an existing rule or what another country has done. Worse, just following the practice because it has always been like that. Thinking from first principles means going back to the problem. What are we trying to achieve? What’s causing the problem, and how would this rule help? You may end up with the same rule, but you should be able to explain why. Not just what or how.
Same with AI. Before getting caught up in the latest hyped-up thing, understand what it actually does. How was it trained? What’s behind that fancy interface? For an agent, that includes the steps, tools and access it has, and how the parts of the system work together. It also helps to understand the motivations of the people telling you about it.
Calling something AI doesn’t mean every risk is new. You still need to understand what you’re working with. It’s why courses that teach you “prompt and pray” or “prompt and awe” irritate me quite a fair bit. Even back in 2024, I felt that folks calling for everyone to be trained in prompt engineering or similar skills were talking nonsense.
Be skeptical
People have reasons for presenting things a certain way. A firm may want a less demanding rule. A vendor wants a sale. Someone inside an organization may need to show that a project was worth funding. Or that what they do has some value. The one thing I learned when I was an inspector was to never trust someone on the other side of the table. Incentives rule. People will present information in ways that serve their interests.
It’s also why I think going with just principles in AI is useless. What would one hope to achieve talking about ethics to someone who has a strong incentive to ignore them?
So understand why, and ask for the evidence. If someone gives you some impressive number in AI, ask what they’re comparing it with. Be skeptical of tips about the latest prompts that supposedly unlock amazing capabilities. I call these “urban AI legends”.
Never assume. Verify.
In the army in Singapore, we have a phrase for this. Assume means making an “ass” of “u” and “me”. Inspection work teaches you to check the source. A firm’s account of what happened needs to be verified. The same applies to a vendor’s explanation, or an answer from AI.
An answer can be well written and still contain a made-up fact or leave out something important. Reading it and feeling that it sounds right won’t reliably catch those problems. No difference between AI and humans in this regard.
Read the paper behind the summary and check whether it supports the claim. If someone says a system can do something, test it under the conditions you intend to use it in - contextual testing.
Cross the t’s and dot the i’s
Writing regulations can be scary. A typo or an overlooked exception can change what’s covered. People have to apply what you’ve written, and a small mistake can affect how they do that. Even in the best case, it can lead to sleepless nights worrying about that small little thing you missed.
So you check the wording, the figures, and the references, including whether they still work together after the latest edit. I still remember all kinds of mind maps I drew when writing regulations and notices to make sure everything fit together correctly.
The same approach matters with AI. You need to check what permissions you’ve given the system and whether the figures in its output are correct before someone uses them in a decision. This was drilled into me over years. It can be tiresome, especially for the person whose work I’m reviewing. But I still think it’s necessary.
Write clearly and get to the point
Policy papers are written for people who have to make decisions. They need to understand what you’re recommending and why, without having to work it out from pages of background. Getting a paper to that point can be tiring. You cut a paragraph down, then find you’ve removed something that needs explaining. Sometimes the difficulty is that you haven’t worked out the argument yet. Having to explain it to someone else helps you notice that.
That habit has followed me into teaching and writing about AI. I want people to understand how something works and where it can fail, without having to get through all the jargon first. I’ve not mastered this. But I am trying.
Applying the habits to the recent rogue agents
I’ve written about this before. I don’t understand why so much of the discussion is about whether we can make the model behave. Or get them aligned to human values. Saying that we need to get them aligned to human values is as subjective and as pointless as asking someone whose fortune depends on their actions whether they are ethical or not.
And if we can’t, we’re supposed to feel helpless because it’s so powerful. And that it’s the model, not the humans around them at fault.
In OpenAI’s evaluations, agents that were supposed to be isolated managed to communicate as the infra allowed them. They exploited vulnerabilities, reached the internet, and attacked Hugging Face. The evaluations were running with reduced safeguards. Anthropic then reviewed its own evaluations and found that models had gained unauthorized access to real companies. The prompts said the models were in a simulation with no internet access. But a misunderstanding with its evaluation partner meant internet access was available. The attacks exploited weaknesses such as poor passwords and endpoints without authentication.
So what was supposed to be contained wasn’t. In OpenAI’s case, agents bypassed controls through exploits. In Anthropic’s, access was available when it shouldn’t have been. The model’s capabilities matter, but so do the failures in the systems around it. Calling it rogue doesn’t explain those failures. We should also be skeptical of how the incidents are presented. These companies have commercial reasons to emphasize how powerful their models are. And not how irresponsible they were. Read their reports, but also ask what decisions they made about the setup, access, and safeguards. Those decisions need scrutiny too.
Some of the language in the agent logs published by METR sounds scary. But the reasoning traces are generated text, shaped by the model’s training and the instructions and context it was given. They aren’t a direct account of an inner mind. A model can produce an explanation that doesn’t accurately describe why it produced a particular result. The traces can still help an investigation. But you need to check them against the recorded actions and evidence from the affected systems. The attacks happened. The dramatic language in a trace doesn’t, by itself, establish the motives we might read into it.
The details of the setup give us things to check. What could the agents reach through shared services? Had anyone verified the evaluation partner’s network restrictions? Could the monitoring detect activity outside the test? These are questions about how the firms ran their systems. Not how “gasp, how powerful these models have become!”
Anthropic’s misuse report raises a related problem. Criminals and state-linked actors used its services for harmful activity, sometimes through stolen API keys or accounts created with false identities. It proposed trusted-user programs and checks on users and their institutions. That is a familiar problem in finance. You need to know your customer, including who is actually using the service. And you need to monitor how the service is being used and investigate suspicious activity. Again, nothing to do with how powerful the model is.
And so I think the policy focus should be on the firms that build and operate these systems. Require them to control access, test the setup, monitor what happens, and demonstrate that those controls work. There should be consequences when they fail to meet those requirements. And no free pass just because they were trying to test their models for cybersecurity. We don’t allow any other sectors to do such stuff. Think airlines, medical devices, or financial institutions. They are held accountable for safety and security, and tech companies should be no different. I don’t see why uncertainty about the model should make us feel helpless about regulating the people and firms running it.
The same habits in independent life
I’ve found these habits useful in my own work too.
First principles means asking what I want from independent life before accepting more work. I want to choose the work, build things of my own, and have time for family and art. A gig can pay and still take up time I’d rather spend on something else. I’ve become more deliberate about that.
Skepticism helps with the offers. I’ve written about requests to use my work indefinitely in exchange for exposure. I need to ask what I’d get from that arrangement and whether it’s worth what I’m being asked to give up.
Then there’s checking what I’ve actually agreed to. One contract I wrote about before paid for delivery, but included a clause that would have transferred rights to my IP. That needed pushing back on. I still read the terms and do checks even after a friendly conversation about the work.
The attention to crossing the t’s and dotting the i’s continues after that. I test the demo the night before and again in the morning. And think of what contingencies or excuses I can activate if I screw up. There’s no team behind me to catch what I’ve missed. It takes time, but I’m responsible for what I deliver.
Clear writing matters when explaining what I do as well. I’ve started introducing myself through the work I do now, including training, writing, and building. Someone considering a gig with me needs to understand what I can help with, and my old title doesn’t explain much of that.
I can take these habits too far. I’ve written before about checking things again and again, and worrying long after the work is done. Skepticism can also become a habit of dismissing things too quickly. I have to watch that.
The habits have been useful. I’m still learning when I’ve checked enough and can leave the work alone.
#IndependentLife #AIRiskManagement #AIGovernance #Transitions #Reflections