Consider the following excerpt from the Alan Agresti's book on generalized linear models:

"Having formed a model matrix $\textbf{X}$ and observed $\textbf{y}$, how do we obtain parameter estimates $\hat{\beta}$ and fitted values $\hat{\mu} = \textbf{X}\hat{\beta}$ that best satisfy the linear model? The standard approach uses the least squares method. This determines the value of $\hat{\mu}$ that minimizes \begin{align*} \lVert\textbf{y} - \hat{\mu}\rVert^{2} = \sum_{i=1}^{n}(y_{i} - \hat{\mu}_{i})^{2} = \sum_{i=1}^{n}\left(y_{i} - \sum_{j=1}^{n}\hat{\beta}_{j}x_{ij}\right)^{2} \end{align*}

That is, the fitted values $\hat{\mu}$ are such that \begin{align*} \lVert\textbf{y} - \hat{\mu}\rVert \leq \lVert\textbf{y} - \mu\rVert\quad\text{for all}\quad\mu\in C(\textbf{X}) \end{align*}

Where $C(\textbf{X})$ denotes the column space of $\textbf{X}$. Using least squares corresponds to maximum likelihood when we add a normality assumption to the model."

Maybe it is a naive question, but I am not able to understand the validity of the last the phrase. Can someone help me out? Thanks in advance!

  • 2
    $\begingroup$ The key is that the same sum of squares enters the likelihood function of $y-\hat{\mu}$ under normality. $\endgroup$ – Megadeth May 5 '19 at 1:24

Under the normality assumption (combined with linearity and homoskedasticity) you have:

$$\mathbf{y} | \mathbf{x} \sim \text{N}(\mathbf{x} \boldsymbol{\beta} ,\sigma^2 \boldsymbol{I}).$$

This gives you the log-likelihood function:

$$\begin{equation} \begin{aligned} \ell_{\mathbf{y}|\mathbf{x}}(\boldsymbol{\beta}, \sigma) &= \ln \text{N}(\mathbf{y}| \mathbf{x} \boldsymbol{\beta} ,\sigma^2 \boldsymbol{I}) \\[6pt] &= - \frac{n}{2} \cdot \ln(2\pi) - n \ln \sigma - \frac{1}{2 \sigma^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 \\[6pt] &= - n \ln \sigma - \frac{1}{2 \sigma^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 + \text{const}. \\[6pt] \end{aligned} \end{equation}$$

Thus, letting $\hat{\sigma}$ be the MLE of $\sigma$ we have:

$$\begin{equation} \begin{aligned} \hat{\boldsymbol{\beta}}_\text{MLE} &= \underset{\boldsymbol{\beta}}{\text{arg max }} \ell_{\mathbf{y}|\mathbf{x}}(\boldsymbol{\beta}, \hat{\sigma}) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max }} \Big( - n \ln \hat{\sigma} - \frac{1}{2 \hat{\sigma}^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 + \text{const} \Big) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max }} \Big( - \frac{1}{2 \hat{\sigma}^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 \Big) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg min }} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 \\[6pt] &= \hat{\boldsymbol{\beta}}_\text{OLS}. \\[6pt] \end{aligned} \end{equation}$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.