3
$\begingroup$

I am running a LASSO regression. What makes me feel strange is that, although $y$ has a very nice bell shape, my prediction $\hat{y}$ is highly skewed to the left, i.e., a positive skew.

I had expected that the predicted value would has a bell shape. I cannot understand where the skewness come from.

I have double-checked the prediction by adding the linear model myself (instead of using the built-in R function). The skewness persists.

The only thing I can think of is, some of my predictors $x_{3}$ and $x_{9}$ are from Poisson process. So these 2 variables are positive only. Both of them are selected by LASSO and have negative $\beta$.

However, I remember that least square regression does not make any assumption on the distribution of predictors.

Can anyone help me to make sense out of this?

Thanks

$\endgroup$
1
  • $\begingroup$ Strong regularization is certainly capable of skewing predictions. Perhaps you want to turn down or even turn off regularization and satisfy yourself that there isn't something else wrong, like you forgot to take the $log()$ of some input varibable. Then turn regularization back on and up, but slowly. $\endgroup$ Commented May 28, 2019 at 20:19

1 Answer 1

4
$\begingroup$

Regularization suppresses the value of each model coefficient individually based on its absolute value (LASSO) or square of its value (Ridge) taking into account a) its role in the unregularized loss function, b) the values of other coefficients in the regularization term and their roles in the loss function, and c) the degree of modulation of regularization by the hyperparameter $\lambda.$ The overall effect is to minimize the regularized loss function, but what it does parameter-by-parameter depends on all of the details.

How could this go possibly wrong?

What if the regularization penalized a fairly relevant term in the model more than the others because the coefficient of that term was high relative to the others and because the regularization itself was over emphasized?

Here is an example. Consider a data generated to follow $y=0.1 x+1.0 x^2 - 1.0 x^4$. In the region $-1<x<1$, $y(x)$ looks like slightly squashed parabola. Outside the region, it begins to be dominated by $x^4$ and it opens down, accounting for the assymetric "frown." If we add Gaussian noise ($\sigma^2=0.01$) the histogram of $y$ will look pretty close to Gaussian, which was your requirement. Here with $n=1000$:

enter image description here

This has a fairly "normal" looking histogram.

enter image description here

If we analyze these data with an $8^{th}-$order polynomial and without the benefit of regularization, we find that all coefficients fail to reject the null hypothesis $\hat{\beta_i}=0, i=0\dots 8$ except for the three we set to non-zero values, and these take on their prescribed values with $p$ values well into the null rejection regime. This unregularized result is perfectly consistent with the synthetic data, where $\beta_i=0, i=0\dots 8$ except for $\beta_1=0.1, \beta_2=1.0$ and $\beta_4=-1$. If you stop here, you have the "right" answer.

If you proceed to LASSO and choose an unfortunate value of $\lambda$, say $\lambda=0.03$ then you may find yourself with a model that has killed $\beta_4$ and made $\beta_1=0.039$ nearly as prominent as $\beta_2=0.043$ (when it should be $10\times$ smaller) and also added an intercept term with $\beta_0=.122$ that wasn't in the original source of the data. And the histogram of $\hat{y}$ will look like this

enter image description here

which is the positive skew in the $\hat{y}$ that you mention in your post.

If you raise $\lambda$ a bit further, your results will be dominated by the regularization and the model will collapse to a constant value.

All of which leads to a few of questions:

  1. Have you thoroughly investigated the effect that $\lambda$ is having on your results?

  2. Do you really need to regularize at all? Do you have enough data and an intelligent enough model to rely on your base loss function alone?

  3. Have you made your analysis as conditional as it can be to meet your R&D goals? In the grand choice between managing data with conditionals or managing data with regularization, I am firmly in the conditioning camp.

Good generalization subject to a condition a) is still good generalization and b) comes completed with documentation and testing, a.k.a., the statement of the condition itself.

$\endgroup$
1
  • $\begingroup$ Great answer... $\endgroup$
    – Glen_b
    Commented May 29, 2019 at 2:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.