I have a bid dataset (3,700,000 obs) and I would like to try multiple regression. I have used biglm library and it is fine. However, I have problems (not enough memory) to produce diagnostic plots. Are there any alternative tests to test the assumptions? Thank you!
$\begingroup$
$\endgroup$
4
-
2$\begingroup$ What kind of diagnostic plots are you tring to create? E.g., you should never try to plot 3.7 M residuals. You should sample about 1 % of the residuals and plot these. Or you can summarize the diagnostic data (histograms, boxplots, ...). $\endgroup$– RolandCommented Jul 16, 2021 at 10:21
-
$\begingroup$ Thank you for your suggestion. I manage to use histograms to check for residuals normality. For heteroscedasticity I use the ncvTest(fit) , and for non- independence of errors the durbinWatsonTest(fit). However, I am not sure how to check for non- linearity. $\endgroup$– IreneCommented Jul 16, 2021 at 10:27
-
2$\begingroup$ It is better to have a model fitting strategy than to posit a simple model and try to find out what went wrong. Unless you know things are linear from previous datasets allow continuous predictors to be nonlinear then there is one less thing to check. See RMS for how to use regression splines, etc. $\endgroup$– Frank HarrellCommented Jul 16, 2021 at 10:51
-
1$\begingroup$ If you're just interested in inference on the conditional mean (such as producing a confidence interval for the regression coefficients), near-normality of residuals isn't particularly consequential; correctness of the model for the conditional mean, heteroskedasticity and dependence are the main issues. $\endgroup$– Glen_bCommented Jul 17, 2021 at 2:43
Add a comment
|