2
$\begingroup$

I am working on a census-related project where I am interested in assigning to everyone a score that estimates the probability they will have a demographic characteristic of interest. In this case, the demographic characteristic of interest (say, having red hair) is quite rare. (Roughly 15,000 out of 1,000,000 people have red hair.)

Though it is easy for me to gather all of the data, due to computational efficiency I am down-sampling my 0s and applying corrections from King & Zeng, 2001, Logistic Regression in Rare Events Data.

In an attempt to predict this rare event, I have a couple of questions:

  1. When conducting logistic regression with rare events, how might you suggest executing feature selection?

  2. Is PCA acceptable to use when dealing with logistic regression? With rare-event-corrected logistic regression? Do you suggest other methods of ensuring independent variables are orthogonal for this type of model?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ +1. Just to clarify: your (1) and (2) ask about completely different things, right? Feature selection means selecting some small subset of features that have good predictive power, whereas PC regression means doing PCA on all the features and then using a small subset of leading PCs as new features. PC regression is one possible way to do regularization; it is not doing any feature selection. Apart from that, perhaps you should mention how many predictors you have. $\endgroup$
    – amoeba
    Commented Oct 21, 2015 at 16:30

1 Answer 1

Reset to default
1
$\begingroup$

1) There is wide agreement that logistic regression doesn't work well with rare event data. The logistic distribution is Gumbel or double exponential distributed and simply isn't heavy-tailed enough. One workaround is to use poisson regression, which does fit rare event data.

2) I'm not sure what you mean by "is PCA acceptable?" One issue with rare event data that standard, Gaussian, linear PCA definitely won't deal with well is sparsity. A number of papers have made recommendations regarding more robust PCA approaches. One of the best is Xie and Xing's robust Cauchy PCA which also reviews many other robust solutions.

http://arxiv.org/abs/1412.6506

Regardless, I still think you should clarify what you mean by "acceptable?"

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Perhaps "advisable" would have been a better term to use. Obviously using a simple linear regression model is not the best choice for fitting data that isn't linear. Similarly, a t-test is not the most appropriate decision for non-Normal data. A t-test is robust to non-Normal data. In using "acceptable," I meant to ask if using PCA was a method that can be applied to this type of modeling or if PCA is ill-advised and another method is more advisable. $\endgroup$
    – Matt Brems
    Commented Oct 21, 2015 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.