4
$\begingroup$

Consider the following excerpt from the Alan Agresti's book on generalized linear models:

"Having formed a model matrix $\textbf{X}$ and observed $\textbf{y}$, how do we obtain parameter estimates $\hat{\beta}$ and fitted values $\hat{\mu} = \textbf{X}\hat{\beta}$ that best satisfy the linear model? The standard approach uses the least squares method. This determines the value of $\hat{\mu}$ that minimizes \begin{align*} \lVert\textbf{y} - \hat{\mu}\rVert^{2} = \sum_{i=1}^{n}(y_{i} - \hat{\mu}_{i})^{2} = \sum_{i=1}^{n}\left(y_{i} - \sum_{j=1}^{n}\hat{\beta}_{j}x_{ij}\right)^{2} \end{align*}

That is, the fitted values $\hat{\mu}$ are such that \begin{align*} \lVert\textbf{y} - \hat{\mu}\rVert \leq \lVert\textbf{y} - \mu\rVert\quad\text{for all}\quad\mu\in C(\textbf{X}) \end{align*}

Where $C(\textbf{X})$ denotes the column space of $\textbf{X}$. Using least squares corresponds to maximum likelihood when we add a normality assumption to the model."

Maybe it is a naive question, but I am not able to understand the validity of the last the phrase. Can someone help me out? Thanks in advance!

$\endgroup$
1
  • 2
    $\begingroup$ The key is that the same sum of squares enters the likelihood function of $y-\hat{\mu}$ under normality. $\endgroup$
    – Yes
    May 5, 2019 at 1:24

1 Answer 1

3
$\begingroup$

Under the normality assumption (combined with linearity and homoskedasticity) you have:

$$\mathbf{y} | \mathbf{x} \sim \text{N}(\mathbf{x} \boldsymbol{\beta} ,\sigma^2 \boldsymbol{I}).$$

This gives you the log-likelihood function:

$$\begin{equation} \begin{aligned} \ell_{\mathbf{y}|\mathbf{x}}(\boldsymbol{\beta}, \sigma) &= \ln \text{N}(\mathbf{y}| \mathbf{x} \boldsymbol{\beta} ,\sigma^2 \boldsymbol{I}) \\[6pt] &= - \frac{n}{2} \cdot \ln(2\pi) - n \ln \sigma - \frac{1}{2 \sigma^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 \\[6pt] &= - n \ln \sigma - \frac{1}{2 \sigma^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 + \text{const}. \\[6pt] \end{aligned} \end{equation}$$

Thus, letting $\hat{\sigma}$ be the MLE of $\sigma$ we have:

$$\begin{equation} \begin{aligned} \hat{\boldsymbol{\beta}}_\text{MLE} &= \underset{\boldsymbol{\beta}}{\text{arg max }} \ell_{\mathbf{y}|\mathbf{x}}(\boldsymbol{\beta}, \hat{\sigma}) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max }} \Big( - n \ln \hat{\sigma} - \frac{1}{2 \hat{\sigma}^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 + \text{const} \Big) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg max }} \Big( - \frac{1}{2 \hat{\sigma}^2} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 \Big) \\[6pt] &= \underset{\boldsymbol{\beta}}{\text{arg min }} ||\mathbf{y} - \mathbf{x} \boldsymbol{\beta}||^2 \\[6pt] &= \hat{\boldsymbol{\beta}}_\text{OLS}. \\[6pt] \end{aligned} \end{equation}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.