# Why normality assumption on linear model implies equivalence between least square estimation and maximum likelihood estimation?

Consider the following excerpt from the Alan Agresti's book on generalized linear models:

"Having formed a model matrix $$\textbf{X}$$ and observed $$\textbf{y}$$, how do we obtain parameter estimates $$\hat{\beta}$$ and fitted values $$\hat{\mu} = \textbf{X}\hat{\beta}$$ that best satisfy the linear model? The standard approach uses the least squares method. This determines the value of $$\hat{\mu}$$ that minimizes \begin{align*} \lVert\textbf{y} - \hat{\mu}\rVert^{2} = \sum_{i=1}^{n}(y_{i} - \hat{\mu}_{i})^{2} = \sum_{i=1}^{n}\left(y_{i} - \sum_{j=1}^{n}\hat{\beta}_{j}x_{ij}\right)^{2} \end{align*}

That is, the fitted values $$\hat{\mu}$$ are such that \begin{align*} \lVert\textbf{y} - \hat{\mu}\rVert \leq \lVert\textbf{y} - \mu\rVert\quad\text{for all}\quad\mu\in C(\textbf{X}) \end{align*}

Where $$C(\textbf{X})$$ denotes the column space of $$\textbf{X}$$. Using least squares corresponds to maximum likelihood when we add a normality assumption to the model."

Maybe it is a naive question, but I am not able to understand the validity of the last the phrase. Can someone help me out? Thanks in advance!

• The key is that the same sum of squares enters the likelihood function of $y-\hat{\mu}$ under normality. – Megadeth May 5 '19 at 1:24

$$\mathbf{y} | \mathbf{x} \sim \text{N}(\mathbf{x} \boldsymbol{\beta} ,\sigma^2 \boldsymbol{I}).$$
\begin{equation} \begin{aligned} \ell_{\mathbf{y}|\mathbf{x}}(\boldsymbol{\beta}, \sigma) &= \ln \text{N}(\mathbf{y}| \mathbf{x} \boldsymbol{\beta} ,\sigma^2 \boldsymbol{I}) \\[6pt] &= - \frac{n}{2} \cdot \ln(2\pi) - n \ln \sigma - \frac{1}{2 \sigma^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 \\[6pt] &= - n \ln \sigma - \frac{1}{2 \sigma^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 + \text{const}. \\[6pt] \end{aligned} \end{equation}
Thus, letting $$\hat{\sigma}$$ be the MLE of $$\sigma$$ we have:
\begin{equation} \begin{aligned} \hat{\boldsymbol{\beta}}_\text{MLE} &= \underset{\boldsymbol{\beta}}{\text{arg max }} \ell_{\mathbf{y}|\mathbf{x}}(\boldsymbol{\beta}, \hat{\sigma}) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max }} \Big( - n \ln \hat{\sigma} - \frac{1}{2 \hat{\sigma}^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 + \text{const} \Big) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max }} \Big( - \frac{1}{2 \hat{\sigma}^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 \Big) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg min }} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 \\[6pt] &= \hat{\boldsymbol{\beta}}_\text{OLS}. \\[6pt] \end{aligned} \end{equation}