What are the assumptions for the distribution of the features for regression models like Lasso regression or Ridge regression? Why is it better to have features with Gaussian distributions?

  • 2
    $\begingroup$ Exactly what assumptions would these be? (Most regression models make no assumptions about distributions of the explanatory variables beyond the minimum necessary to assure that solutions exist. In fact, in many such models the explanatory variables are not assumed to be realizations of any random variables at all: they may be determined by the experimenter, for instances.) $\endgroup$ – whuber Dec 1 '14 at 23:04

Your question is not entirely clear. This answer assumes that by "distribution of the features" you mean the conditional distributions of the response across explanatory variables.

Under the General Linear Model regression estimates are obtained by minimising the Residual Sum of Squares

$RSS = \sum\limits_{i=1}^{n} \Big(y_i - \beta_0 - \sum\limits_{j=1}^{p} \beta_j x_{ij} \Big)^2$.

In contrast, Ridge Regression aims to minimise

$RSS + \lambda \; \sum\limits_{j=1}^p \beta_j^2$,

and Lasso Regression aims to minimise

$RSS + \lambda \; \sum\limits_{j=1}^p |\beta_j|$,

where $\lambda$ is a tuning parameter.

So, Ridge Regression and Lasso Regression are special cases of the General Linear Model. They add penalty terms but otherwise all of the same conditions apply, including conditionally independent Gaussian residuals with zero mean and constant variance across the range of the explanatory variable(s).

  • 2
    $\begingroup$ Do You have any sources to back your final statement up? I have not found consistent consensus on this fact, and would like to learn more. Thanks $\endgroup$ – theforestecologist Jul 28 '15 at 21:39

From a Bayesian standpoint, the assumptions are simply in the priors on the coefficients. Ridge regression is equivalent to using a Gaussian prior, whereas LASSO is equivalent to using a Laplace prior. As @whuber said, these models don't make assumptions on the distribution of the explanatory variables.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.