6
$\begingroup$

I'm in a setting where I am trying to model a continuous output variable given ~100 variables and ~100k datapoints. The signal-to-noise ratio is extremely low, and colinearity is very high. Among the variables are many "needle-in-a-haystack" binary-valued features. A "needle-in-a-haystack" binary-valued feature, $f$, is one where $Pr[f==1]$ is small (~0.01), but where it is important for our model to be unbiased when $f==1$.

When I use OLS, the resultant model is properly unbiased when $f==1$. However, the model has undesirable characteristics stemming from noise and colinearity.

When I try elastic-net regularization, the noise/colinearity problems go away. However, it appears that the act of regularizing causes the model to disregard bias for the needle-in-a-haystack features. Even when $f$ is selected by the model, the model generates unacceptably large residuals when $f==1$.

I'm wondering how I can get the best of both worlds. I am currently training an elastic-net regularized model first, and then training a second OLS model to predict the residuals from the needle-in-haystack features. This seems to work decently, but I'm wondering if there is a more standard way.

Improve this question
$\endgroup$
8
  • $\begingroup$ By "unbiased," you mean unbiased predictions, correct? $\endgroup$
    – alex
    Aug 23, 2013 at 17:24
  • $\begingroup$ yes, unbiased predictions $\endgroup$
    – dshin
    Aug 23, 2013 at 17:31
  • $\begingroup$ You may try to implement weighting of observations giving more weight to rare class. Or add rare class observations using sampling with replacement of available rare class objects. These ways the misproportion in size of classes can be diminished. $\endgroup$
    – O_Devinyak
    Aug 23, 2013 at 18:43
  • $\begingroup$ Conversely, you can down-sample your negative cases $\endgroup$
    – David Marx
    Aug 23, 2013 at 20:50
  • 1
    $\begingroup$ @kjetilbhalvorsen Done. $\endgroup$
    – dshin
    Aug 28, 2018 at 14:55

1 Answer 1

Reset to default
0
$\begingroup$

Ultimately, I ended up abandoning regularized approaches, as they are simply too biased for unbalanced categorical/binary features. I've actually grown quite skeptical of regularization in general, at least in my problem domain.

Instead, I went with OLS using stepwise feature selection with k-fold cross-validation. I had to implement some nontrivial machinery to make this work at scale, as the full $n \times k$ matrix doesn't fit in memory.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ You abandoned regularization in favour of stepwise? Interesting, did you read stats.stackexchange.com/questions/250155/… or stats.stackexchange.com/questions/20836/… ? What is your field? Maybe you have many important interactions! Did you have interaction terms in the model? Maybe try randomforest? $\endgroup$
    – kjetil b halvorsen ♦
    Aug 28, 2018 at 18:02
  • $\begingroup$ Yes, there are interaction terms in the model. Random forests are difficult to make fast at query time. Predictions that take longer than ~1 microsecond to compute are not so useful. My domain is strange in that I only act on predictions far from the mean, meaning that I only care about model accuracy on a tiny % of the population. So sparse features that tend to yield outliers are very valuable, but regularization is terrible at handling such features. $\endgroup$
    – dshin
    Aug 28, 2018 at 18:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.