2020

GPT-3 and the Scaling Hypothesis

OpenAI releases GPT-3 — 175 billion parameters, trained on hundreds of billions of tokens. The model itself is impressive, but the paper that accompanies it is the real bombshell: capability emerges from scale alone. No new architecture, no clever tricks — just a bigger Transformer trained on more data. Few-shot learning works. Translation works. Coding works. The implication is staggering: if you keep scaling, you keep getting more general intelligence, almost for free. Every frontier lab pivots within months. The race that produces ChatGPT, Claude, and Gemini begins here, with a single graph showing capability rising smoothly with compute.