Least SquaresLeast Squares

On the first night of the nineteenth century, the Italian astronomer Giuseppe Piazzi spotted a faint new object: Ceres, the first known asteroid. He tracked it for forty-one days before it vanished into the sun's glare. Astronomers across Europe had a puzzle on their hands: where would it reappear?

A 24-year-old Carl Friedrich Gauss computed an orbit from the sparse, noisy observations, and in December 1801 Ceres was found almost exactly where he predicted. Gauss later said his secret was a technique he had used since 1795: choose the answer that minimizes the sum of the squared errors between prediction and observation.

But Gauss had never published it. The first person to put least squares in print was Adrien-Marie Legendre, in an 1805 appendix to a book on comet orbits. When Gauss claimed priority in 1809, Legendre was furious, and the dispute soured relations between two of the era's greatest mathematicians for years.

Why squares rather than, say, absolute errors? Squaring punishes large mistakes disproportionately, produces a unique answer, and, as Gauss showed, is the optimal choice when errors follow the bell curve that now bears his name. It also makes the mathematics tractable: the best fit can be found by solving a system of linear equations.

Least squares became the foundation of regression analysis, the everyday tool of economics, biology, and engineering for two centuries. Fitting a line through data points, the canonical first exercise in any statistics course, is least squares in its simplest form.

The idea scales up astonishingly well. Training a neural network by minimizing squared error on its predictions is a direct descendant of the Ceres calculation: a loss function, a set of parameters, and an algorithm that adjusts them to make errors as small as possible.