We live in an era where Large Language Models (LLMs) can write functional code in seconds, draft eloquent essays, and pass the bar exam. Yet, if you hand these same cutting-edge models a basic Excel spreadsheet and ask them to predict a simple outcome—a task machine learning algorithms mastered decades ago—they suddenly struggle with the absolute basics.

Why does an AI that understands the nuances of quantum physics struggle to predict whether a customer will churn based on a 20-column CSV file?

This is the core question tackled in the fascinating 2026 research paper, Why Large Language Models Fail at Tabular Prediction by Marta Garnelo and Wojciech M. Czarnecki.

Instead of trying to build a better prompt or a complex AI agent to fix the problem, the researchers did something refreshingly simple: they stripped away all the fancy tools and tested a frontier LLM (Claude Opus 4.6) in its purest form to figure out exactly where its brain short-circuits.

Here is a breakdown of what they found, why it matters, and what it means for anyone working with data.

Busting the Myths: What Isn’t Breaking the AI?

Before finding the real culprit, the researchers, Garnelo and Czarnecki (2026) tested four common theories that data practitioners usually blame when LLMs fail at tabular data.

To make this concrete, imagine you are trying to predict if a house will sell above asking price based on a dataset.

Myth 1: The data is too messy or overlapping (Separability)

• The Theory: LLMs fail because real-world data is noisy. The line between a house that sells high and one that doesn’t isn’t clear-cut.

• The Verdict: False. The researchers artificially separated the data so the answers were painfully obvious. Traditional algorithms immediately got 100% accuracy. The LLM barely improved (Garnelo & Czarnecki, 2026).

Myth 2: LLMs can’t read “columns” in text format (Serialization)

• The Theory: When you convert a spreadsheet into a giant block of comma-separated text (CSV), the AI loses the visual structure of columns and rows, getting lost in the “haystack.”

• The Verdict: False. They literally put the correct answer right inside one of the columns as a test. Even with 60 other distracting columns, the LLM found the answer perfectly. It can read the columns just fine (Garnelo & Czarnecki, 2026).

Myth 3: Decimals confuse the AI (Numeric Tokenization)

• The Theory: Numbers like 3.14159265 get chopped up into weird text fragments (tokens) by the AI, confusing its ability to do math.

• The Verdict: False. They rounded all the numbers to just one decimal place. The LLM’s performance didn’t improve at all (Garnelo & Czarnecki, 2026).

Myth 4: We are asking for too much at once (Batch Size)

• The Theory: Asking the AI to predict 100 rows in a single prompt spreads its “thinking power” too thin.

• The Verdict: False. Asking it to predict just a few rows at a time yielded the exact same poor results (Garnelo & Czarnecki, 2026).

The Real Culprit: The Curse of Dimensionality

If it’s not the format, the math, or the messiness, what is it? Dimensionality.

Dimensionality is just a fancy word for “the number of columns (features) in your dataset.”

Garnelo and Czarnecki (2026) found that if you give an LLM a dataset with only 2 columns (e.g., just Square Footage and Number of Bedrooms), it actually performs brilliantly. In fact, in a 2D space, the LLM mathematically acts almost exactly like a classic “K-Nearest Neighbors” algorithm. It looks at the grid, finds similar points close to each other, and makes a highly accurate, logical guess.

But as you add more columns—Year Built, Distance to Schools, Roof Type, Property Tax—the LLM starts to collapse.

By the time the dataset hits around 16 columns, the LLM’s accuracy drops to at or below what you’d get by just guessing the majority class — essentially a coin flip. Meanwhile, 50-year-old algorithms like Logistic Regression or Random Forests stay perfectly flat or actually improve because they use the extra information to make better predictions.

The Analogy: Think of the LLM like a juggler. Juggling two balls (2 dimensions) is easy; they can track the exact trajectory of both. But toss in 16 balls at once, and they don’t just drop the new ones—they drop all of them. The LLM fundamentally lacks the internal wiring to geometrically process high-dimensional spaces in a single pass.

Don’t Trust the AI’s Explanations

One of the most human and dangerous traits of an LLM is its ability to confidently explain its reasoning.

When the researchers asked the LLM why it made certain predictions on the data, the results were alarming. Only 6 of 20 tasks had both the prediction and the stated explanation actually match the real pattern in the data. In 4 more, the prediction was right but the explanation was wrong — a confident, plausible-sounding rule that had nothing to do with how the answer was actually reached. In the remaining 10, neither the prediction nor the explanation matched. Add it up: 14 of 20 tasks — 70% — had explanations that didn’t reflect what the model actually did.

It wasn’t reliably following its own stated rules. Often, it was just generating text that sounded like a good explanation.

Takeaway

This paper is a vital reality check. We are often tempted to throw LLMs at every problem because they are incredibly powerful and easy to use. But AI is not a monolith, and language models are fundamentally wired for sequential text, not multidimensional geometry.

If you are building data pipelines, analyzing user behavior, or running predictive models on spreadsheets:

1. Stick to the Classics: Do not replace your Random Forests, Gradient Boosting (XGBoost), or classical machine learning models with an LLM prompt for tabular prediction. The old tools still win this fight handily.

2. Beware of AI Confidence: Never trust an LLM’s written explanation of how it analyzed raw numbers. The “reasoning” is often a hallucination disconnected from the actual computation.

3. Use LLMs for What They Do Best: Let the LLM write the Python code to build your XGBoost model, or have it summarize the text-based results. Just don’t ask it to do the math itself.

The AI space is flooded with flashy new technologies, making it incredibly tempting to throw an LLM at every problem. But sometimes, the shiniest tool in the toolbox isn’t a replacement for a simple, sturdy hammer. When dealing with structured tabular data, classical machine learning models often predict better, execute faster, and save a massive amount of tokens. Ultimately, it comes down to choosing the right tool for the data at hand. Before defaulting to the latest state-of-the-art LLMs, it might be worth asking: could a traditional model get this job done just as efficiently?

References

Garnelo, M., & Czarnecki, W. M. (2026). Why large language models fail at tabular prediction. arXiv. https://arxiv.org/pdf/2608.02412