Abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

The paper that sparked the AI revolution — introduces the Transformer, now the backbone of GPT, Claude, Gemini, and virtually every modern language model.

Abstract

We explore how generating a chain of thought — a series of intermediate reasoning steps — significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple prompting technique.

Asking an AI to 'think step by step' dramatically improves its reasoning — and this paper proved it rigorously across math, logic, and common-sense benchmarks.

Abstract

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.

More data + more compute + bigger models = predictably better AI — this paper quantified the exact relationship and permanently changed how AI labs operate.