I have large dataset (around 2 million records and 300 features) with a lot of missing data. Most of the independent variables are categorical (some of these variables have more than 40 valid values). The outcome is either Y or N. The Y outcome is a rare event: around 98% of outcomes are N.
I'm supposed to fit a logistic regression model to these data. I took random sample of them, keeping the same distribution. I am working in R, but I'm new to both R and logistic regression modeling and I have some questions:
factorto the outcome. Do I need to apply it on every categorical independent variable? I have more than 200 variables, some of them have only 2 valid values while others have 40! Will it affect the size of the data?
Is there any advice about attribute selection? Should it be done before fitting the logistic regression model or after, depending on the results?
Is it recommended to take biased sample data where the outcome Y is more than the original distribution in the large data?
There are fields like
groupId, etc. What type of data do we consider these to be? How to deal with them?
What other predictive models are suitable for this kind of data?