6
$\begingroup$

These are the linear regression assumptions:

Linearity: The relationship between X and the mean of Y is linear. Homoscedasticity: The variance of residual is the same for any value of X. Independence: Observations are independent of each other. Normality: For any fixed value of X, Y is normally distributed.

Do they apply to Lasso Regression, too?

$\endgroup$
6
  • 3
    $\begingroup$ LASSO as a machine learning algorithm has no assumptions. You only get into assumptions if you want optimality guarantees, which is common in statistics $\endgroup$
    – Firebug
    Dec 25, 2022 at 18:02
  • $\begingroup$ Does this apply to all algorithms? $\endgroup$
    – Antonio
    Dec 26, 2022 at 1:03
  • $\begingroup$ @Antonio What do you mean? $\endgroup$
    – Dave
    Dec 26, 2022 at 1:25
  • $\begingroup$ As in, I only get into assumptions if I want optimal results @Dave $\endgroup$
    – Antonio
    Dec 26, 2022 at 1:28
  • 1
    $\begingroup$ @Antonio An algorithm doesn’t care about much of anything. You can shoehorn absolute garbage into a LASSO or even OLS and get some kind of output, though the utility is questionable. $\endgroup$
    – Dave
    Dec 26, 2022 at 1:52

3 Answers 3

10
$\begingroup$

I will be a contrarian and say that most assumptions do not apply to LASSO regression.

In the classical linear model, those assumptions are used to show that the OLS estimator is the minimum-variance linear unbiased estimator (Gauss-Markov theorem) and to have correct t-stats, F-stats, and confidence intervals.

In LASSO regression, those are less important. LASSO gives a biased estimator, so any of the Gauss-Markov business for the minimum-variance linear unbiased estimator no longer applies; indeed, LASSO is not even a linear estimator of the coefficients, so Gauss-Markov doubly does not apply. Then the hypothesis testing and confidence intervals would not be of primary concern for a LASSO regression. Indeed, it does not even seem agreed upon how such inference would be performed.

(Linearity still matters, but that’s because OLS and LASSO both estimate the same parameters, which are the coefficients on the features.)

Overall, the typical assumptions are assumed in an OLS linear regression because of nice properties of later inferences that are not particularly important for a situation where LASSO regression would be applied.

$\endgroup$
9
$\begingroup$
  1. Normality is not an assumption of linear regression.

  2. Yes, they do. Lasso regression is a linear regression with a penalty term on the magnitude of the coefficients; the penalty term in no way affects the structure of the underlying model (linearity, independence, homoskedasticity) and the assumptions are the same.

$\endgroup$
2
$\begingroup$

Yes, they are valid also for Lasso, Ridge Regression and Elastic Net. I think you are referring to classical linear model (CLM) assumptions for cross-sectional regression:

  1. Linear in Parameters
  2. Random Sampling
  3. No perfect collinearity
  4. Zero Conditional Mean
  5. Homoskedasticity

where the first four are used to establish unbiasedness of OLS, whereas the fifth is employed to derive the usual variance formulas and to conclude that OLS is Best linear unbiased.

One can also assume normality of the error term to obtain the exact sampling distribution of t statistics and F statistics, so that one can carry out exact hypotheses tests, but normality is not a necessary assumption for CLM.

$\endgroup$
1
  • $\begingroup$ Does Lasso have any impact on assumption number 3 (perfect collinearity)? $\endgroup$ Dec 24, 2022 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.