1
$\begingroup$

I'm reading here that LASSO makes no particular assumption concerning the distribution of residuals, which somehow confuses me.

Consider a linear regression problem with 1 target $y$, $P$ features $\mathbf{x}$ and a training set $\mathcal{D}=(X,Y)$. When looking at this supervised learning problem through discriminative modelling glasses, we are basically postulating the conditional pdf of the latent DGP has a certain form $ p(y \vert \mathbf{x}, \mathcal{D}) $, which we will try to infer `as best as we can' from the data at hand.

Opting for a frequentist approach, we more specifically look for `optimal' model parameters $\mathcal{\theta}^*$ which will allow us to write $p(y\vert\mathbf{x},\mathcal{D})=p(y\vert\mathbf{x},\theta^*(\mathcal{D}))$ given our linear modelling assumption: $$ y = \mathbf{\theta}^T \mathbf{x} + \epsilon \tag{1} $$

When using LASSO, the above model is said to be penalised in the sense that we look for the `optimal' parameter set $$ \theta_{\text{LASSO}} = \arg \min_\theta \Vert Y - X\theta\Vert_2^2 + \lambda \Vert \theta \Vert_1 \tag{2} $$ in an effort to trade in some of the estimator's bias to decrease its variance (decreasing model complexity).

The reason I am interested in the distributional assumptions of $\epsilon$ is because I would like to write out the distribution $p(y \vert \mathbf{x},\theta_{\text{LASSO}})$ which should IMO directly come from $(1)$, our discriminative modelling assumption.

Because the LASSO estimate $(2)$ can be seen as the MAP estimate of model $(1)$ assuming

  • Gaussian white noise $\epsilon$ (hence a Gaussian likelihood model)
  • Laplace prior on the model weights $\theta$

I would be inclined to think that if I assume residuals to be Gaussian in $(2)$, I stay consistent with the choices I made for my estimator of $\theta$, which would lead to a Gaussian $p(y\vert\mathbf{x},\theta_{\text{LASSO}})$.

However, this seems to be in conflict with the idea that LASSO makes no distributional assumptions on residuals here.

Can someone help me understand where I got things mixed up? Of course the problem is the same for ridge regression. Hence the general title of this question. As a side question, if no assumption is made, what would be a good way to estimate the latent distribution of our model residuals: non-parametrically using sample residuals observed on the training set?

| cite | improve this question | | | | |
$\endgroup$
2
$\begingroup$

I think that things get mixed up when you make the assumption of Gaussian residuals. Please note that I'm not much of a Bayesian, so I can't speak to the necessity of Gaussian residuals for the LASSO estimate to be the maximum a posteriori (MAP) estimate of the linear regression with a Laplace prior on model weights $\theta$, as you note in the question.

But Gaussian residuals are certainly not required for ordinary least squares (OLS) to provide the best linear unbiased estimates (BLUE) of the $\theta_i$ in your equation (1). Under the Gauss-Markov theorem all you need to assume about residuals for OLS to provide BLUE are that residuals have zero mean, are homoscedastic, and are uncorrelated. So even standard frequentist OLS doesn't need to assume Gaussian residuals; such residuals just help in some frequentist statistical tests.

LASSO as expressed in your equation (2) similarly need not make any assumptions about the residuals to provide potentially useful penalized estimates of the $\theta_i$. In practice a range of values for $\lambda$ is evaluated with cross-validation or other resampling to provide a final model (including the choice of $\lambda$) that provides a particular bias/variance tradeoff. But equation (2) can certainly be solved over a range of $\lambda$ values without any assumption about the distribution of the residuals. The distribution of the residuals might, however, affect how useful those estimates are.

For working out the distribution $p(y \vert \mathbf{x},\theta_{\text{LASSO}})$ you certainly are welcome to assume Gaussian residuals, but neither ordinary nor penalized regressions require such a strict assumption.

| cite | improve this answer | | | | |
$\endgroup$
  • $\begingroup$ Thank you for your answer, especially for pointing me be back to the OLS estimator, which could itself be seen as the result of a Bayesian approach assuming (i) Gaussian residuals (ii) uninformative prior on model weights (e.g. $p(\theta)=\alpha>0$). I understand now that my confusion came from the fact that I regarded OLS/LASSO/Ridge as "models". But they have nothing to do with the model $y = \theta^T \mathbf{x} + \epsilon$ and simply represent methods to estimate the coefficients $\theta$ of a given model, regardless of $\epsilon$. This is the gist of your last sentence right? $\endgroup$ – Quantuple Aug 9 '17 at 8:49
  • $\begingroup$ @Quantuple you summarized my last sentence perfectly well. $\endgroup$ – EdM Aug 9 '17 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.