Is there a method for regression (my variable Y has binomial distribution ) when n>>p ? I have a data set with 350000 observations and 4 variables. I am using R (I also using SAS).
some help would be appreciated
Partially answered in comments:
What's wrong with using regular logistic regression when you have a lot of data? I don't see any problem here. – gung
Heck, with so many observations and only 4 variables, why not go completely nonparametric? – generic_user
Tell us a little bit more about your goals. Do you need an interpretable model? Then go for logistic regression, as suggested by @gung (actually, in this kind of problem I would always start with l.r., to have some kind of baseline). Do you need to maximize predictive accuracy? Nonparametric approaches such as neural networks, random forest, SVM, gradient boosting, etc. may work. If you give us more details on the problem you can get a more specific suggestion. With only 4 features I don't think you'll ever get great results, though (unless it's a very simple problem). – DeltaIV
The OP comments, showing that the real problem maybe is linear separation, leading to fitted probabilities of 0 or 1 (which will not generalize well to new data ...) How to deal with (quasi)separation is answered here: How to deal with perfect separation in logistic regression?
My data set have 259 000 observations and the binary variable Y=1 has low occurrence. How can we deal with that ? – OP
Finally, a classic paper on learning with unbalanced data sets. – DeltaIV