1
$\begingroup$

Can you please check my work in the problem below?


Consider a location model:

$$X_i =\theta +e_i, \quad i=1,\ldots,n $$

where $e_1, e_2, \ldots e_n $ are iid with pdf $f(z)$. Define $\mathbf{\mu}=\theta \mathbf{1}$ where $\mathbf{1}$ is a vector with all its components equal to $1$. Let $V$ be the subspace of vectors of the form $\mathbf{\mu}$. Then

$$\mathbf{X}=\mathbf{\mu}+\mathbf{e} \quad \mathbf{\mu} \in V $$

It makes sense therefore to estimate $\mathbf{\mu}$ by a vector in $V$ which is "closest" to $\mathbf{X} $, i.e.

$$\mathbf{\hat{\mu}}=\mathrm{Argmin} || \mathbf{X}-\mathbf{v} ||,\quad \mathbf{v} \in V $$

If the error PDF is the standard Laplace, show that the minimization of the above is equivalent to maximizing the likelihood when the norm is the $l1$ norm given by

$$||v||_1=\sum_{i=1}^n | v_i | $$


Of all the components of the location model, the error is the random one so it would make sense to make the one-to-one transformation $e_i=x_i-\theta$ in the PDF of $e_i$. The likelihood then becomes:

$$L \left( \theta \right) =\frac{1}{2^n} exp \{ -\sum |x_i -\theta | \} $$

and the log likelihood

$$ l \left( \theta \right) =-n \log2 -\sum_{i=1}^n |x_i -\theta | $$

Now it seems very clear that the maximization of $ -\sum_{i=1}^n |x_i -\theta |$ is equivalent to the minimization of $\sum_{i=1}^n |X_i-\theta| $ as desired. Does that suffice or am I required to do more?

Thanks in advance.

$\endgroup$
1
  • 1
    $\begingroup$ Your work looks fine. It can even be performed in reverse to show that minimizing the $L^1$ norm of the residuals is uniquely associated with the Laplace family of distributions in maximum likelihood models. $\endgroup$
    – whuber
    Dec 29 '13 at 21:02
1
$\begingroup$

Just like the OLS estimator is equivalent to the MLE in the linear regression model with normally distributed errors, the least absolute deviations estimator [LAD] estimator is equivalent to the MLE in the linear regression model with Laplace distributed errors.

Linear regression model

Consider the linear regression model $$ \boldsymbol{Y}_i = \boldsymbol{X}_i'\boldsymbol{\beta} + \varepsilon_i $$ $i=1, \ldots, n$.

OLS estimator

The OLS estimator is defined as $$ \hat{\boldsymbol{\beta}} = \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n \left(\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right)^2 $$

LAD estimator

The LAD estimator is $$ \tilde{\boldsymbol{\beta}} = \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n \left|\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right| $$

Log-likelihood of the linear regression model

The log-likelihood for the linear regression model can be defined as $$ \log L(\boldsymbol{\beta} \mid \left\{\boldsymbol{Y}_i , \boldsymbol{X}_i\right\}_{i=1}^n) = \sum_{i=1}^n\log f(\varepsilon_i \mid \boldsymbol{\beta}) $$

1. MLE for linear regression model with Normal errors

$$ \begin{align} \hat{\boldsymbol{\beta}}_{MLE} &= \arg\max_{\boldsymbol{\beta}}\sum_{i=1}^n\log \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{1}{2}\left(\frac{\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}}{\sigma}\right)^2\right) \end{align} $$

Ignoring terms that don't depend on $\boldsymbol{\beta}$ and simplifying, we get $$ \begin{align} \hat{\boldsymbol{\beta}}_{MLE} &= \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n \left(\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right)^2 \\ &= \hat{\boldsymbol{\beta}} \end{align} $$

2. MLE for linear regression model with Laplace errors

$$ \begin{align} \tilde{\boldsymbol{\beta}}_{MLE} &= \arg\max_{\boldsymbol{\beta}}\sum_{i=1}^n\log \frac{1}{2\upsilon}\exp\left(-\left|\frac{\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}}{\upsilon}\right|\right) \end{align} $$ As before, ignoring terms that do not depend on $\boldsymbol{\beta}$ and simplifying, we get $$ \begin{align} \tilde{\boldsymbol{\beta}}_{MLE} &= \arg\min_{\boldsymbol{\beta}}\sum_{i=1}^n\left|\boldsymbol{Y}_i -\boldsymbol{X}_i'\boldsymbol{\beta}\right| \\ &= \tilde{\boldsymbol{\beta}} \end{align} $$

$\endgroup$
2
  • $\begingroup$ Yes indeed. These are cases of $l1$ and $l2$ norms and minimizing the quantities is equivalent to maximising their negative. $\endgroup$
    – JohnK
    Dec 29 '13 at 20:55
  • $\begingroup$ @JohnK Quite so. $\endgroup$ Dec 29 '13 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.