Simple Linear Regression

Simple linear regression fits $Y=\alpha +\beta X+\epsilon$ by choosing $\alpha ,\beta$ to minimize the sum of squared residuals. The resulting slope has the closed form $\beta =\text{Cov}(X,Y)/\text{Var}(X)$, and the intercept is $\alpha =E[Y]-\beta E[X]$.

Residuals ${\hat{\epsilon }}_i=y_i-{\hat{y}}_i$ represent the variation in $Y$ not explained by $X$. By construction, the least-squares residuals are uncorrelated with $X$ and average to zero.

$\beta$ measures how much $Y$ is expected to change per unit change in $X$; it depends on the units and scale of both variables, unlike correlation which is unitless.