Geometry and AI. What do they have to do with each other?

I have built and audited models, both AI and non-AI. But the term ‘geometry’ very rarely appears next to these models.

Now, I’ve designed transformer models that are able to learn graph structures that make the most sense for a specific prediction. So I have always known that models can learn some form of structure but this diptych of two papers shows something quite fascinating.

One was shared with me by the geometry guru Agus Sudjianto, and the other I serendipitously chanced upon, straight after reading Agus’ paper.

📖 Left Panel: "Deep sequence models tend to memorize geometrically"

We usually view model predictions as something that comes from associations. A→B, B→C and so on and so forth.

This paper found that even after models learned associations, they still naturally go on to find what the paper calls geometric memory. Instead of A→B, B→C, they want to learn A→C. Or even A→Z. Even when it takes 100x the number of steps to learn this geometric memory. Somehow, geometric patterns emerge from the learning process.

It’s like learning a new city. Home → coffee shop one day. Coffee shop → office another. Now you know the way home from the office.

📖 Right Panel: "BLADE: Bivector-Driven Logical Adaptive Decoding"
Geometry can help with confused AI too.

The paper uses three basic geometric concepts to think about a model's internal state:

Scalar: "Is this path compliant?"
Vector: "Where is this reasoning heading?"
Bivector: "How much tension between competing paths?"

When the bivector is high, branch and verify. When it's flat, let the model proceed.

Works as a triage method to filter out what’s more important to focus on. I also liked the way the paper applied this to ‘stressed’ states: conjunction; disjunction; exception; nested negation etc. A taxonomy of how logic trips up AI.

Same city. Picking between two 7-Elevens a block apart? Just pick one. Don't think too much. Choosing between two alleys that look similar? One is a shortcut, the other leads to a dead end after a long walk. Think twice. And harder.

One paper explains the natural emergence of geometry. The other uses geometry for control.

I need to go and brush up my geometric math.

#AI #AIRiskManagement #Geometry