Geometric Interpretation of Maximum Likelihood in a location model

Question

Can you please check my work in the problem below?

---

Consider a location model:

$$X_i =\theta +e_i, \quad i=1,\ldots,n $$

where $e_1, e_2, \ldots e_n $ are iid with pdf $f(z)$. Define $\mathbf{\mu}=\theta \mathbf{1}$ where $\mathbf{1}$ is a vector with all its components equal to $1$. Let $V$ be the subspace of vectors of the form $\mathbf{\mu}$. Then

$$\mathbf{X}=\mathbf{\mu}+\mathbf{e} \quad \mathbf{\mu} \in V $$

It makes sense therefore to estimate $\mathbf{\mu}$ by a vector in $V$ which is "closest" to $\mathbf{X} $, i.e.

$$\mathbf{\hat{\mu}}=\mathrm{Argmin} || \mathbf{X}-\mathbf{v} ||,\quad \mathbf{v} \in V $$

If the error PDF is the standard Laplace, show that the minimization of the above is equivalent to maximizing the likelihood when the norm is the $l1$ norm given by

$$||v||_1=\sum_{i=1}^n | v_i | $$

---

Of all the components of the location model, the error is the random one so it would make sense to make the one-to-one transformation $e_i=x_i-\theta$ in the PDF of $e_i$. The likelihood then becomes:

$$L \left( \theta \right) =\frac{1}{2^n} exp \{ -\sum |x_i -\theta | \} $$

and the log likelihood

$$ l \left( \theta \right) =-n \log2 -\sum_{i=1}^n |x_i -\theta | $$

Now it seems very clear that the maximization of $ -\sum_{i=1}^n |x_i -\theta |$ is equivalent to the minimization of $\sum_{i=1}^n |X_i-\theta| $ as desired. Does that suffice or am I required to do more?

Thanks in advance.

Answer 1

Just like the OLS estimator is equivalent to the MLE in the linear regression model with normally distributed errors, the least absolute deviations estimator [LAD] estimator is equivalent to the MLE in the linear regression model with Laplace distributed errors.

Linear regression model

Consider the linear regression model $$ \boldsymbol{Y}_i = \boldsymbol{X}_i'\boldsymbol{\beta} + \varepsilon_i $$ $i=1, \ldots, n$.

OLS estimator

The OLS estimator is defined as $$ \hat{\boldsymbol{\beta}} = \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n \left(\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right)^2 $$

LAD estimator

The LAD estimator is $$ \tilde{\boldsymbol{\beta}} = \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n \left|\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right| $$

Log-likelihood of the linear regression model

The log-likelihood for the linear regression model can be defined as $$ \log L(\boldsymbol{\beta} \mid \left\{\boldsymbol{Y}_i , \boldsymbol{X}_i\right\}_{i=1}^n) = \sum_{i=1}^n\log f(\varepsilon_i \mid \boldsymbol{\beta}) $$

1. MLE for linear regression model with Normal errors

$$ \begin{align} \hat{\boldsymbol{\beta}}_{MLE} &= \arg\max_{\boldsymbol{\beta}}\sum_{i=1}^n\log \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{1}{2}\left(\frac{\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}}{\sigma}\right)^2\right) \end{align} $$

Ignoring terms that don't depend on $\boldsymbol{\beta}$ and simplifying, we get $$ \begin{align} \hat{\boldsymbol{\beta}}_{MLE} &= \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n \left(\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right)^2 \\ &= \hat{\boldsymbol{\beta}} \end{align} $$

2. MLE for linear regression model with Laplace errors

$$ \begin{align} \tilde{\boldsymbol{\beta}}_{MLE} &= \arg\max_{\boldsymbol{\beta}}\sum_{i=1}^n\log \frac{1}{2\upsilon}\exp\left(-\left|\frac{\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}}{\upsilon}\right|\right) \end{align} $$ As before, ignoring terms that do not depend on $\boldsymbol{\beta}$ and simplifying, we get $$ \begin{align} \tilde{\boldsymbol{\beta}}_{MLE} &= \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n\left|\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right| \\ &= \tilde{\boldsymbol{\beta}} \end{align} $$