Shallow, Deep, Strong and Weak AI

I seem to be drawn to what is behind the surface lately. In art, and otherwise.

Some of my watercolors of the back of things. I seem to be drawn to what is behind the surface lately. In art, and otherwise.

I realise that discussions on AI that just focus on just prompt and use cases frustrate me. Not just a little.

Two meetings this week with very different views of prompts and use cases in AI somehow reminded me of the issues I had with a strong/general and weak/narrow framing of AI I encountered when looking into the AIGP.

Packed week. Exhausted at the end.

A meeting to review training courses. Facilitating a roundtable discussing learning materials and guides. A talk for hundreds of folks from a bank’s group risk management. And a full day class for insurers. Along the way, getting into H2O AI’s 100 list, a former Bloomberg reporter published a piece on AI in finance with my view on the AIRG, and plenty of discussions on gigs and training.

At the end of the week, strangely, the thought that stuck with me was how much discussions on AI that just focus on just prompt and use cases irk me. Especially when coupled with dismissals of the need to understand AI fundamentals.

Weak and Strong AI

It starts with the AIGP certification, one of the main credentials in AI governance. I remember reading a study guide a year ago, and seeing a point on distinguishing strong/general AI, and weak/narrow AI. And thinking that this framing made no sense. Probably a reason why I cannot bring myself to take the AIGP up to today. Nothing to do with the quality of the certificate itself, but the impression from that specific encounter.

Calling narrow AI weak makes no sense from an AI governance angle. Narrow AI is not necessarily weak. It may even be able to beat an LLM at the one thing it was trained to do handily. And weak and narrow AI can also do a lot of harm. Discriminatory hiring filters, flawed credit models, facial recognition that gets the wrong person arrested. These are all non general purpose AI. Are they weak?

I find it shallow because it sorts AI by a weird and superficial line. Which brings me to the next point on shallow AI.

Shallow and Deep AI

This came to mind because of two meetings in the past week that touched on prompts and use cases.

The first meeting was to review the suitability of some AI courses. But the meeting seemed to be overly focused on narrow use cases with fancy names, as well as prompts and more prompts.

The beauty of AI today is that we no longer need to keep to narrowly defined use cases, but can think about innovative ways to perform different tasks with AI. And prompts no longer are the only determinant of the quality of AI outputs. Someone at the meeting wondered why the need to focus on such things that AI could do. I echoed his thought in another way later in the meeting.

In the second, we were developing a set of guides for a role. We started with a two page prompt and were supposed to brainstorm and edit it. At the end of the 2 hour discussion that I facilitated, the entire prompt was deleted and we had a more structured guide that talked about the limitations of AI, the need for clear contexts tailored for different orgs, how to judge what is good, and how to build human capabilities while using AI.

Much much better. I felt happier at the end of the discussion than the start.

The Prompt Spiral

This is the thing I keep running into. Prompt and pray, and prompt and awe.

Prompt and pray is typing something in and trusting whatever comes back. Or trusting that the LLM will always follow your instructions and getting surprised when it does not.

Prompt and awe is being dazzled by a clever prompt, a slick-looking output, or just blindly applying a prompt to a fancy sounding use case and mistaking that for understanding.

A fixation on better prompts and cleverer use case names, and barely anything on what’s underneath them, is not a good one for improving AI fluency.

I have seen it enough to have a name for the pattern.

The prompt spiral.

You keep refining the prompt without understanding the system. The output improves and you do not know why. The work seems to look good. But you never even define what good means. Nothing wrong happens. But nothing deep happens either. And then one day something breaks, you blame the prompt, and your only solution is to continue to adjust the darn prompt. And down the rabbit hole one goes.

What I think matters more

What I think matters more is looking through the surface to the thing underneath.

A prompt is shallow. A use case with a clever name is shallow.

Underneath every AI system sits a short list of questions that goes beyond the prompt and the use case. You need to know what the task really is, once you strip off the fancy use case name. You need to know what data the system runs on, and whether that data is any good. You need to know what the model is actually doing, which really comes down to the task it performs, because what counts as good, and how it fails, both depend on that task. You need to know how it will fail, and whether you would even notice when it does. And in the end, you have to understand it, learn to work with it, know how to evaluate what it produces, and own the risk of what comes out.

And if you can do these, you realise that thinking about any use case, from any domain, is just a matter of assembly.

Get those right and the prompt is a very small detail. Get them wrong and no prompt will save you. This is why I think treating “prompt design” and “prompt engineering” as core AI skills is problematic and fragile.

The good discussions I have been in, the second meeting this week included, spend their time down there, on the task and the data and the failure modes.

The frustrating ones stay up on the surface, polishing prompts and inventing fancier and fancier names for use cases, and call it an AI strategy.

It relates to something I learned a long time ago and have written about since. Systems thinking. Understand the parts, understand how they interact, understand the whole.

We get into trouble the moment we treat anything complex as one big black box and just poke at it from the outside.

But maybe it’s just me. I see this so often that sometimes I wonder if I am the crazy one.

#AI #AIFluency #AIGovernance #AILiteracy #Reflections #AIRiskManagement