Laws That Break: Where Large Language Model Scaling Expectations Fail
We used to think artificial intelligence was simple arithmetic. You want a smarter model? Throw more money at it. More compute, bigger datasets, larger parameter counts-the math promised that performance would improve in a smooth, predictable curve. This belief, known as scaling laws, is empirical rules predicting how AI model performance improves with increased compute, data, and size, became the industry’s holy grail. It allowed companies to forecast returns on investment with startling accuracy.
But reality has a habit of breaking neat equations. Since the initial discovery of these laws, researchers have repeatedly found that they shatter under specific conditions. The assumption that "bigger is always better" or that "compute scales linearly with quality" is no longer valid. In fact, following the old scaling laws blindly can lead to massive waste, unstable models, and security vulnerabilities. Here is where the expectations fail and what you need to know to navigate the new landscape of AI development.
The Chinchilla Shock: When Bigger Was Actually Worse
To understand why scaling laws break, we have to look at the moment the first major crack appeared. For years, the dominant wisdom came from Kaplan et al. (2020), which suggested that loss decreased predictably as a power law with model size, dataset size, and compute. The implication was clear: if you had a fixed budget, you could trade off these variables somewhat freely.
Then came DeepMind’s 2022 paper on Chinchilla, a 70-billion parameter language model optimized for compute efficiency. This study didn’t just tweak the formula; it overturned it. Researchers discovered that the previous scaling laws were fundamentally wrong regarding optimal resource allocation. The industry had been building models that were too big and undertrained.
DeepMind trained Chinchilla using the exact same amount of computational power as their previous 280-billion parameter model, Gopher. However, instead of making the model smaller, they kept the parameters high but drastically increased the training data-from 300 billion tokens for Gopher to 1.4 trillion tokens for Chinchilla. The result? Chinchilla outperformed both Gopher and OpenAI’s 175-billion parameter GPT-3, despite being significantly smaller than its competitors.
This finding established a new rule: for every increase in compute, both model size and data size should scale by approximately the same proportional amount. If you increase compute by 10x, you should make the model ~3.1x bigger and the dataset ~3.1x larger. If you ignore this balance, you are wasting resources. Building a massive model without proportionally increasing the data leads to diminishing returns and poor generalization.
The Production Trap: Why Optimal Training Isn't Enough
If Chinchilla optimality is the gold standard for training efficiency, why don’t all companies follow it strictly? Because there is a critical gap between minimizing training loss and achieving production-quality inference. This is where the first scaling law breaks down in practice.
Chinchilla optimality focuses on reaching the lowest possible loss during training with a fixed compute budget. But in the real world, users care about how the model performs after training is complete. They care about reasoning, creativity, and robustness during inference. To achieve this, researchers often deliberately "overtrain" models-feeding them far more data than Chinchilla’s formula suggests.
Meta’s LLaMA models are a prime example. They were explicitly overtrained to ensure better performance during deployment. Subsequent studies have shown that scaling laws in this "overtraining regime" require dataset sizes up to 32x larger than what Chinchilla optimality predicts. If you strictly follow the efficient training laws, your model might be computationally cheap to build but mediocre when actually used. The law holds for the lab, but fails for the product.
Reinforcement Learning: The Wild West of Scaling
Pretraining large language models follows rigorous, empirically validated power laws. Reinforcement Learning (RL) does not. As models move from pretraining to alignment via RL (such as Reinforcement Learning from Human Feedback, or RLHF), the predictable scaling relationships vanish almost entirely.
In pretraining, the relationship between compute and performance is stable. In RL, high variance in per-token policy ratios causes training instability. A single token can dominate the loss expression, leading to numerical crashes or erratic behavior. This issue is particularly acute when training on long sequences or using Mixture-of-Experts (MoE) architectures.
Because traditional scaling laws don’t apply here, researchers cannot predict how much compute is needed to align a model safely. Best practices are often anecdotal, dependent on specific hyperparameters, and non-generalizable. Companies must test RL design choices the hard way-by running expensive, large-scale experiments and seeing what works. This creates a major bottleneck, limiting iteration speed and hindering technical progress in one of the most critical areas of AI safety and capability.
Safety Doesn't Scale Linearly
Perhaps the most dangerous failure of scaling laws occurs in adversarial contexts. We often assume that as a model becomes more capable, its safety mechanisms scale proportionally. This is false. Jailbreak scaling laws follow entirely different mathematical relationships than benign capability scaling.
Research modeling proxy language as a spin-glass system reveals that adversarial prompt-injection attacks can amplify attack success rates from slow polynomial growth to exponential growth with the number of inference-time samples. Short injected prompts act like weak magnetic fields, yielding power-law scaling. Long injected prompts act like strong magnetic fields, producing exponential scaling.
This means that safety properties do not scale predictably alongside capability improvements. A model that is 10% smarter might be 100% easier to jailbreak if the scaling dynamics of its vulnerability space are exponential while its defensive capabilities are only polynomial. This disconnect creates a perpetual friction between theory and application, requiring separate, domain-specific validation for safety rather than relying on general scaling trends.
Test-Time Compute: Shifting the Frontier
A new paradigm is emerging that challenges the very definition of "scaling." Traditionally, we scaled training compute. Now, we are scaling test-time compute. This involves applying more computational resources at inference time to improve accuracy, enabling reasoning models to perform multiple passes to work through complex problems.
This approach, often seen in systems like o1 or other reasoning-focused models, requires intensive amounts of accelerated computing. The scaling relationships for inference-time computation differ fundamentally from training-time allocations. Efficiency frontiers shift when moving from optimizing for training loss to optimizing for deployment accuracy. This suggests that the future of AI scaling isn't just about bigger models, but about smarter use of compute during the actual interaction with the user.
| Scaling Domain | Predictability | Key Failure Mode | Practical Implication |
|---|---|---|---|
| Pretraining (Kaplan) | High | Assumes infinite data availability | Good for forecasting baseline performance |
| Chinchilla Optimality | High (for training loss) | Fails for production inference quality | Requires overtraining for real-world use |
| Reinforcement Learning | Low | High variance and instability | No reliable ROI prediction; trial-and-error required |
| Adversarial Safety | None | Exponential vulnerability growth | Safety must be tested independently of capability |
| Test-Time Compute | Moderate | High latency costs | Shifts cost from training to inference |
What This Means for Your Strategy
The repeated discovery of broken scaling laws suggests a meta-pattern: scaling laws hold precisely only for narrowly-defined objectives, typically training loss in pretraining with unlimited data. Real-world constraints-finite data, inference requirements, safety considerations, and RL instability-systematically violate laboratory predictions.
For developers and strategists, this means you can no longer rely on a single universal formula. You must validate scaling assumptions within your specific domain. If you are building a recommendation system, Netflix has shown that quality follows predictable power laws. If you are aligning a chatbot, expect chaos. If you are concerned about jailbreaks, expect exponential risks.
The path forward involves recognizing that scaling is not a monolith. It is a collection of distinct, sometimes contradictory, relationships. By understanding where these laws break, you can avoid wasted compute, build safer models, and allocate resources where they actually matter.
What are LLM scaling laws?
LLM scaling laws are empirical formulas that predict how the performance of a large language model improves as you increase three factors: model size (parameters), dataset size (tokens), and computational resources. Originally proposed by Kaplan et al., they suggest that loss decreases in a predictable power-law fashion.
Why did the Chinchilla paper change everything?
The Chinchilla paper revealed that previous scaling laws were suboptimal. It showed that for every unit of compute, model size and data size should scale equally. Previously, companies were building models that were too large and undertrained. Chinchilla proved that a smaller model trained on more data outperforms a larger model trained on less data, given the same compute budget.
Do scaling laws apply to Reinforcement Learning?
No, scaling laws for Reinforcement Learning (RL) are highly unpredictable. Unlike pretraining, RL suffers from high variance and instability. There are no established power laws that reliably predict performance gains from increased compute in RL, making it a trial-and-error process.
Why is safety scaling different from capability scaling?
Safety vulnerabilities, such as jailbreaks, can scale exponentially with inference samples, while capabilities often scale polynomially. This means a model can become significantly more vulnerable to attacks even as its overall intelligence improves modestly, breaking the assumption that safety scales linearly with capability.
What is test-time scaling?
Test-time scaling refers to using additional computational resources during the inference phase (when the model answers questions) rather than just during training. This allows models to "think" longer, perform multiple reasoning steps, and produce higher-quality outputs, shifting the cost structure of AI deployment.
- Jul, 16 2026
- Collin Pace
- 0
- Permalink
- Tags:
- LLM scaling laws
- Chinchilla optimality
- AI training efficiency
- reinforcement learning scaling
- adversarial safety
Written by Collin Pace
View all posts by: Collin Pace