# Geometric Interpretation of Maximum Likelihood in a location model

Can you please check my work in the problem below?

Consider a location model:

$$X_i =\theta +e_i, \quad i=1,\ldots,n$$

where $e_1, e_2, \ldots e_n$ are iid with pdf $f(z)$. Define $\mathbf{\mu}=\theta \mathbf{1}$ where $\mathbf{1}$ is a vector with all its components equal to $1$. Let $V$ be the subspace of vectors of the form $\mathbf{\mu}$. Then

$$\mathbf{X}=\mathbf{\mu}+\mathbf{e} \quad \mathbf{\mu} \in V$$

It makes sense therefore to estimate $\mathbf{\mu}$ by a vector in $V$ which is "closest" to $\mathbf{X}$, i.e.

$$\mathbf{\hat{\mu}}=\mathrm{Argmin} || \mathbf{X}-\mathbf{v} ||,\quad \mathbf{v} \in V$$

If the error PDF is the standard Laplace, show that the minimization of the above is equivalent to maximizing the likelihood when the norm is the $l1$ norm given by

$$||v||_1=\sum_{i=1}^n | v_i |$$

Of all the components of the location model, the error is the random one so it would make sense to make the one-to-one transformation $e_i=x_i-\theta$ in the PDF of $e_i$. The likelihood then becomes:

$$L \left( \theta \right) =\frac{1}{2^n} exp \{ -\sum |x_i -\theta | \}$$

and the log likelihood

$$l \left( \theta \right) =-n \log2 -\sum_{i=1}^n |x_i -\theta |$$

Now it seems very clear that the maximization of $-\sum_{i=1}^n |x_i -\theta |$ is equivalent to the minimization of $\sum_{i=1}^n |X_i-\theta|$ as desired. Does that suffice or am I required to do more?

• Your work looks fine. It can even be performed in reverse to show that minimizing the $L^1$ norm of the residuals is uniquely associated with the Laplace family of distributions in maximum likelihood models.
– whuber
Dec 29 '13 at 21:02

Just like the OLS estimator is equivalent to the MLE in the linear regression model with normally distributed errors, the least absolute deviations estimator [LAD] estimator is equivalent to the MLE in the linear regression model with Laplace distributed errors.

## Linear regression model

Consider the linear regression model $$\boldsymbol{Y}_i = \boldsymbol{X}_i'\boldsymbol{\beta} + \varepsilon_i$$ $i=1, \ldots, n$.

## OLS estimator

The OLS estimator is defined as $$\hat{\boldsymbol{\beta}} = \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n \left(\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right)^2$$

The LAD estimator is $$\tilde{\boldsymbol{\beta}} = \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n \left|\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right|$$

## Log-likelihood of the linear regression model

The log-likelihood for the linear regression model can be defined as $$\log L(\boldsymbol{\beta} \mid \left\{\boldsymbol{Y}_i , \boldsymbol{X}_i\right\}_{i=1}^n) = \sum_{i=1}^n\log f(\varepsilon_i \mid \boldsymbol{\beta})$$

### 1. MLE for linear regression model with Normal errors

\begin{align} \hat{\boldsymbol{\beta}}_{MLE} &= \arg\max_{\boldsymbol{\beta}}\sum_{i=1}^n\log \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{1}{2}\left(\frac{\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}}{\sigma}\right)^2\right) \end{align}

Ignoring terms that don't depend on $\boldsymbol{\beta}$ and simplifying, we get \begin{align} \hat{\boldsymbol{\beta}}_{MLE} &= \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n \left(\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right)^2 \\ &= \hat{\boldsymbol{\beta}} \end{align}

### 2. MLE for linear regression model with Laplace errors

\begin{align} \tilde{\boldsymbol{\beta}}_{MLE} &= \arg\max_{\boldsymbol{\beta}}\sum_{i=1}^n\log \frac{1}{2\upsilon}\exp\left(-\left|\frac{\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}}{\upsilon}\right|\right) \end{align} As before, ignoring terms that do not depend on $\boldsymbol{\beta}$ and simplifying, we get \begin{align} \tilde{\boldsymbol{\beta}}_{MLE} &= \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n\left|\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right| \\ &= \tilde{\boldsymbol{\beta}} \end{align}

• Yes indeed. These are cases of $l1$ and $l2$ norms and minimizing the quantities is equivalent to maximising their negative. Dec 29 '13 at 20:55
• @JohnK Quite so. Dec 29 '13 at 20:58