Normal Equations for Ordinary Least Squares

Least squares minimizes $\Vert y-X\beta {\Vert }^2$ over $\beta$.

Differentiate the squared residual norm with respect to $\beta$ and set the gradient to zero.

1. 1. Differentiate the objective

   The squared error $\Vert y-X\beta {\Vert }^2$ has gradient $-2X^T(y-X\beta )$ with respect to $\beta$.

2. 2. Set the gradient to zero

   Setting the gradient to zero and rearranging gives the normal equations.

   $$X^{T}X\beta =X^{T}y$$

Answer: X^T X beta = X^T y