Compositional Generalization in NLP: Can LLMs Reason Systematically?

You type a command into an AI chatbot, something simple like "turn on the blue lamp twice." The model does it perfectly. Then you ask it to "turn off the red chair around three times," and it freezes or hallucinates. Why? This is the core of compositional generalization: the ability to understand new combinations of known parts using learned rules. It’s not about memorizing every possible sentence; it’s about applying grammar and logic to things the model has never seen before.

For years, researchers asked if Large Language Models (LLMs) could truly reason systematically like humans, or if they were just fancy autocomplete engines. The answer isn't a simple yes or no. By 2026, we know that while modern LLMs are incredibly powerful, their systematicity is often brittle. They can ace standard tests but collapse when faced with novel structures. Let’s break down what this means for developers, researchers, and anyone building AI applications.

The Core Problem: Memorization vs. Compositionality

Human language learning is compositional. If you know what "jump" means and what "twice" means, you automatically understand "jump twice," even if you’ve never heard that specific phrase before. You combine primitives (words) with rules (grammar) to create meaning. This is called systematicity. It allows us to handle infinite novelty with finite knowledge.

Neural networks, however, learn differently. They often rely on statistical correlations found in training data. Standard sequence-to-sequence models, including early Transformers, achieved near-perfect accuracy (96-99%) on random test splits where test sentences looked similar to training ones. But when researchers created strict compositional splits, performance plummeted to 16-35%. This gap reveals that models were memorizing surface patterns rather than learning underlying rules.

Benchmarks That Expose the Gap

To measure this, researchers built synthetic datasets designed to be unforgiving. These aren’t natural texts scraped from the web; they are generated by formal grammars to ensure specific gaps between training and testing.

SCAN (Simple Contextual Action Navigation), introduced in 2018, is the classic example. It maps commands like "walk left twice" to action sequences. The tricky part? The test set might contain actions or modifiers never seen in training, or combinations of them. If a model only saw "run twice" but never "jump twice," can it figure out "jump twice"? Early models failed hard here.

Then came CFQ (Compositional Freebase Questions) in 2020. This dataset uses real-world knowledge bases (Freebase) and asks questions like "Who directed the movie starring Tom Hanks?" CFQ measures compound divergence-how different the training and test distributions are. High divergence means the test requires combining concepts in ways unseen during training. Baseline models showed steep accuracy declines as divergence increased, proving they weren't truly parsing the semantic structure.

COGS (Compositional Generalization Challenge Based on Semantic Interpretation) takes this further by pairing sentences with formal logical forms. It deliberately excludes certain syntactic structures in training, such as transitive uses of verbs that only appeared intransitively. To pass COGS, a model must infer the rule for transitivity without explicit examples. In 2020, top models scored only 16-35% on these generalization sets, despite scoring over 96% on standard tests.

How LLMs Changed the Game

Between 2022 and 2025, the narrative shifted. Researchers discovered that you don’t always need to retrain a model to improve its systematicity. Instead, you can guide it through prompting.

Techniques like Least-to-Most Prompting decompose complex problems into sub-problems. For example, instead of asking an LLM to solve a complex math word problem directly, you prompt it to first identify the variables, then the operations, and finally compute the result. When applied to compositional benchmarks, this approach allowed LLMs to achieve 95-99.7% accuracy on SCAN and CFQ using less than 1% of the original training data.

This suggests that LLMs possess latent compositional abilities that are unlocked by structured instructions. The model doesn't need to memorize the specific combination; it needs to be reminded to apply the correct rule step-by-step. However, this success is fragile. It works best when the decomposition strategy matches the task structure. If the prompt is vague, the model reverts to pattern matching.

Where LLMs Still Fail

Despite these wins, new benchmarks released in 2024 and 2025 show that systematicity remains a weak point for abstract reasoning.

Consider the STaR Benchmark (Systematic Relational Reasoning). Unlike SCAN, which deals with syntax, STaR tests relational logic. Can the model infer a hidden rule connecting objects? Many popular LLMs perform above chance but fail to reach high accuracy, indicating they struggle to apply abstract relational rules consistently.

Another frontier is Thematic Generalization. Here, models must identify a narrow latent theme from a few examples. For instance, if shown images of "small red circles" and "large blue squares," can they identify that "medium green triangles" fit the theme of "geometric shapes with color and size attributes"? Current models often over-generalize, picking broader categories that miss the fine-grained rule. This shows that while LLMs are good at linguistic composition, they lack robust conceptual compositionality.

A critical insight from recent studies is the concept of critical complexity. Every model has a threshold of reasoning depth beyond which it stops generalizing and starts memorizing. Smaller or older models hit this wall quickly. Frontier models push the boundary further, but it still exists. This means that as your application gets more complex, you cannot assume the LLM will scale its reasoning indefinitely.

Practical Implications for Developers

If you are building an NLP application, how do you mitigate these risks?

• Don't trust in-distribution metrics: If you fine-tune a model, create a custom validation set that contains novel combinations of entities and structures. Random splits lie.
• Use Chain-of-Thought (CoT): Explicitly prompting the model to explain its steps helps enforce systematic processing. It forces the model to attend to intermediate states rather than jumping to conclusions based on surface cues.
• Control Decoding Parameters: Surprisingly, sampling strategies matter. Studies show that top-p (nucleus) sampling can improve compositional accuracy by several percentage points compared to greedy decoding. Greedy decoding often locks onto frequent local patterns, ignoring rare but correct compositional paths.
• Monitor Distribution Shifts: Compositional failures spike when input length increases or when new entity types are introduced. Test your system against longer inputs and unfamiliar vocabulary before deployment.

The Future: Structure vs. Scale

Is the solution more parameters? Not necessarily. While scaling helps, many experts argue that true systematicity requires stronger inductive biases. This could mean integrating explicit syntactic parsers, using graph neural networks for relational tasks, or developing hybrid architectures that combine symbolic logic with neural flexibility.

We are moving away from the question "Can LLMs reason?" toward "Under what conditions do LLMs reason systematically?" The evidence up to mid-2026 supports a nuanced view: LLMs can approximate human-like compositionality in controlled settings, especially with advanced prompting. But they remain brittle under distribution shifts and complex relational demands. For now, treat compositional generalization as an active engineering challenge, not a solved feature.

• Sep, 14 2026
• Collin Pace
• 0
• Permalink

• Tags:
• compositional generalization
• large language models
• systematic reasoning
• NLP benchmarks
• SCAN dataset

Written by Collin Pace

• 

  View all posts by: Collin Pace