5
$\begingroup$

What are the assumptions for the distribution of the features for regression models like Lasso regression or Ridge regression? Why is it better to have features with Gaussian distributions?

$\endgroup$
1
  • 2
    $\begingroup$ Exactly what assumptions would these be? (Most regression models make no assumptions about distributions of the explanatory variables beyond the minimum necessary to assure that solutions exist. In fact, in many such models the explanatory variables are not assumed to be realizations of any random variables at all: they may be determined by the experimenter, for instances.) $\endgroup$
    – whuber
    Dec 1, 2014 at 23:04

2 Answers 2

9
$\begingroup$

Your question is not entirely clear. This answer assumes that by "distribution of the features" you mean the conditional distributions of the response across explanatory variables.

Under the General Linear Model regression estimates are obtained by minimising the Residual Sum of Squares

$RSS = \sum\limits_{i=1}^{n} \Big(y_i - \beta_0 - \sum\limits_{j=1}^{p} \beta_j x_{ij} \Big)^2$.

In contrast, Ridge Regression aims to minimise

$RSS + \lambda \; \sum\limits_{j=1}^p \beta_j^2$,

and Lasso Regression aims to minimise

$RSS + \lambda \; \sum\limits_{j=1}^p |\beta_j|$,

where $\lambda$ is a tuning parameter.

So, Ridge Regression and Lasso Regression are special cases of the General Linear Model. They add penalty terms but otherwise all of the same conditions apply, including conditionally independent Gaussian residuals with zero mean and constant variance across the range of the explanatory variable(s).

$\endgroup$
1
  • 3
    $\begingroup$ Do You have any sources to back your final statement up? I have not found consistent consensus on this fact, and would like to learn more. Thanks $\endgroup$ Jul 28, 2015 at 21:39
6
$\begingroup$

From a Bayesian standpoint, the assumptions are simply in the priors on the coefficients. Ridge regression is equivalent to using a Gaussian prior, whereas LASSO is equivalent to using a Laplace prior. As @whuber said, these models don't make assumptions on the distribution of the explanatory variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.