4
$\begingroup$

I have large dataset (around 2 million records and 300 features) with a lot of missing data. Most of the independent variables are categorical (some of these variables have more than 40 valid values). The outcome is either Y or N. The Y outcome is a rare event: around 98% of outcomes are N.

I'm supposed to fit a logistic regression model to these data. I took random sample of them, keeping the same distribution. I am working in R, but I'm new to both R and logistic regression modeling and I have some questions:

  1. I applied factor to the outcome. Do I need to apply it on every categorical independent variable? I have more than 200 variables, some of them have only 2 valid values while others have 40! Will it affect the size of the data?

  2. Is there any advice about attribute selection? Should it be done before fitting the logistic regression model or after, depending on the results?

  3. Is it recommended to take biased sample data where the outcome Y is more than the original distribution in the large data?

  4. There are fields like userId, groupId, etc. What type of data do we consider these to be? How to deal with them?

  5. What other predictive models are suitable for this kind of data?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
9
$\begingroup$

Q1: For Y/N variables you can, but it won't make any difference except give you control over whether Y or N is your base category in the default model fitting. For 40 category variables your model matrix will end up pretty big, it's true. More importantly it will require a lot of data to fit. Combinatorially speaking you need information about all combinations of independent variables, and even with the data you have, there'll be a lot of interpolation and model assumption.

Q2: The machine learning folk may have some ideas here. I dimly remember something about chi-squared and mutual information measures for selecting variables. It's also possible you could get the model fitting process to do it by using a Lasso (L1 regularization, a.k.a Laplace prior) on the coefficients, although I'm not sure how well current implementations scale.

Q3: If you take a biased sample then you can do a classic rare events design analysis. King and Zheng, 2001 is a good resource for how to do so: it's very simple and amounts to a simple intercept correction. So yes, this is a good idea - just don't forget to correct for the sampling scheme.

Q4: User ids are potential grouping variables, so you could, if you wanted aggregate data according to user (or other group). That could also make the estimation problem easier by moving from Bernoulli to Binomial assumptions about the dependent variable.

Q5: Any classification model will do, frankly: support vector machines, decision trees, or anything else should work, provided they scale to the size of your data and/or you can apply the rare events correction to them. Regularized logistic regression is a good start though. You might find the literature on text classification a good place to start looking.

Improve this answer
$\endgroup$
6
  • $\begingroup$ @Conjugate Prior Thank you for your answer. I'm sorry if my questions could be basic but as I mentioned I'm new to this. ForQ1: so in the case of 40 categorical values or more, I didn't understand what is the recommended step. grouping the values? for what level? $\endgroup$
    – simplyme
    Apr 18, 2012 at 16:43
  • $\begingroup$ and for Q4: can you clarify it more please. like in R, how it deals with it when I apply the logistic function using new_id as IV, like if I apply this code in R: mydata.logr <- glm(mydata ~ new_id, family=binomial("logit")) , will it consider it a numeric or as nominal since it's id not really numeric value, and in both cases does it affect the model accuracy? $\endgroup$
    – simplyme
    Apr 18, 2012 at 16:56
  • $\begingroup$ and sorry another beginner Q: in the previous code i put only one independent variable. how will be the code if I want the model to include all the IV in mydata? i know it will be like mydata.logr <- glm(mydata ~ new_id + mailing_id + Age, family=binomial("logit")) Is there another way to write it to include all the fields in the data? $\endgroup$
    – simplyme
    Apr 18, 2012 at 17:02
  • $\begingroup$ The data-shrinking version of Q4 is to aggregating over id, so the new dependent variable for some id would be [times id says Y, times id says N] treat it as binomial. The independent variables would be some average of id's independent variables. This may not make sense in your case and would be a last resort if the data was just too big. $\endgroup$
    – conjugateprior
    Apr 18, 2012 at 17:12
  • $\begingroup$ See the glm help page for how to treat something as binomial. essentially you regress on the new two column dependent variable I described above (usually constructed using cbind) when you run the model. $\endgroup$
    – conjugateprior
    Apr 18, 2012 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.