I am working on a linear model problem,
where $X$ has around 200 columns and around 300K rows
not surprisingly, I am using LASSO to bring down the complexity of the model.
but when I put the model into production, I noticed some problem. Some features, e.g., feature 104 is very infrequent. Among 300K samples, it only shows up <1000 times. In all other samples, this feature is silent (i.e., equals zero)
However, when this feature shows up, it really has some impact on the model: it either says some significant features (e.g., feature 15) suddenly becomes irrelevant ($\beta$ should be zero), or some significant features' $\beta$ should be of opposite sign.
LASSO tends to 'discriminate' against such infrequent or 'sparse' features because assign $\beta$ to them is not economical. LASSO tends to allocate features to those features who show up all the time and have some impact on $y$ all the time (this is my understanding, not necessarily correct)
I think the fundamental problem here is that:
The underline distribution where samples are pulled is a multi-Gaussian distribution. Sparse features are a warning sign saying that 'we are now under a different Gaussian'
My question is:
Is there any well-established regression mechanism that can handle such problem?