# Needle-in-a-haystack Regularized Regression

I'm in a setting where I am trying to model a continuous output variable given ~100 variables and ~100k datapoints. The signal-to-noise ratio is extremely low, and colinearity is very high. Among the variables are many "needle-in-a-haystack" binary-valued features. A "needle-in-a-haystack" binary-valued feature, $f$, is one where $Pr[f==1]$ is small (~0.01), but where it is important for our model to be unbiased when $f==1$.

When I use OLS, the resultant model is properly unbiased when $f==1$. However, the model has undesirable characteristics stemming from noise and colinearity.

When I try elastic-net regularization, the noise/colinearity problems go away. However, it appears that the act of regularizing causes the model to disregard bias for the needle-in-a-haystack features. Even when $f$ is selected by the model, the model generates unacceptably large residuals when $f==1$.

I'm wondering how I can get the best of both worlds. I am currently training an elastic-net regularized model first, and then training a second OLS model to predict the residuals from the needle-in-haystack features. This seems to work decently, but I'm wondering if there is a more standard way.

• By "unbiased," you mean unbiased predictions, correct? – alex Aug 23 '13 at 17:24
• yes, unbiased predictions – dshin Aug 23 '13 at 17:31
• You may try to implement weighting of observations giving more weight to rare class. Or add rare class observations using sampling with replacement of available rare class objects. These ways the misproportion in size of classes can be diminished. – O_Devinyak Aug 23 '13 at 18:43
• Conversely, you can down-sample your negative cases – David Marx Aug 23 '13 at 20:50
• @kjetilbhalvorsen Done. – dshin Aug 28 '18 at 14:55

Instead, I went with OLS using stepwise feature selection with k-fold cross-validation. I had to implement some nontrivial machinery to make this work at scale, as the full $n \times k$ matrix doesn't fit in memory.