I am working on a linear model problem,

$y =f(X)$

where $X$ has around 200 columns and around 300K rows

not surprisingly, I am using LASSO to bring down the complexity of the model.

but when I put the model into production, I noticed some problem. Some features, e.g., feature 104 is very infrequent. Among 300K samples, it only shows up <1000 times. In all other samples, this feature is silent (i.e., equals zero)

However, when this feature shows up, it really has some impact on the model: it either says some significant features (e.g., feature 15) suddenly becomes irrelevant ($\beta$ should be zero), or some significant features' $\beta$ should be of opposite sign.

LASSO tends to 'discriminate' against such infrequent or 'sparse' features because assign $\beta$ to them is not economical. LASSO tends to allocate features to those features who show up all the time and have some impact on $y$ all the time (this is my understanding, not necessarily correct)

I think the fundamental problem here is that:

The underline distribution where samples are pulled is a multi-Gaussian distribution. Sparse features are a warning sign saying that 'we are now under a different Gaussian'

My question is:

Is there any well-established regression mechanism that can handle such problem?


  • $\begingroup$ It depends on what "0" actually means. Is the feature ordinal or categorical? Does the "0" represent a missing value, or is it a real number, e.g., a count or measurement that actually has the value 0 for those samples? $\endgroup$ – jbowman Mar 4 '18 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.