I'm in a setting where I am trying to model a continuous output variable given ~100 variables and ~100k datapoints. The signal-to-noise ratio is extremely low, and colinearity is very high. Among the variables are many "needle-in-a-haystack" binary-valued features. A "needle-in-a-haystack" binary-valued feature, $f$, is one where $Pr[f==1]$ is small (~0.01), but where it is important for our model to be unbiased when $f==1$.

When I use OLS, the resultant model is properly unbiased when $f==1$. However, the model has undesirable characteristics stemming from noise and colinearity.

When I try elastic-net regularization, the noise/colinearity problems go away. However, it appears that the act of regularizing causes the model to disregard bias for the needle-in-a-haystack features. Even when $f$ is selected by the model, the model generates unacceptably large residuals when $f==1$.

I'm wondering how I can get the best of both worlds. I am currently training an elastic-net regularized model first, and then training a second OLS model to predict the residuals from the needle-in-haystack features. This seems to work decently, but I'm wondering if there is a more standard way.

  • By "unbiased," you mean unbiased predictions, correct? – alex Aug 23 '13 at 17:24
  • yes, unbiased predictions – dshin Aug 23 '13 at 17:31
  • You may try to implement weighting of observations giving more weight to rare class. Or add rare class observations using sampling with replacement of available rare class objects. These ways the misproportion in size of classes can be diminished. – O_Devinyak Aug 23 '13 at 18:43
  • Conversely, you can down-sample your negative cases – David Marx Aug 23 '13 at 20:50
  • Just of out of curiosity: Have you tried SCAD (smoothly clipped absolute deviation - Fan & Li, 2001) penalization? It might have this "best of both worlds" property you are looking concerning your problem. – usεr11852 Aug 23 '13 at 22:44

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.