1
$\begingroup$

I am looking for good materials to learn more about analyzing very large data sets, say 5 million observations, with a reasonable computational time (say less than 10h).

My main interests is traditional analysis methods, such as linear, Poisson and logistic regression, both mixed and not-mixed. Together with simple illustration and presentation of data, that provide easy understandable interpretation and options for confounder correction. I am open to alternative methods yielding reasonable similar results with the methods mentioned above, but reducing the computational time. For example, in some cases using a robust sandwich estimator can account for incorrect specified correlation structure but reduce computational time compared with a cluster methods.

I am familiar with the programs R, SAS and STATA, and would prefer methods available in these programs.

If you know of relevant (English/Danish) courses held by universities, I would love to have links for course descriptions for this as well.

I hope to hear from you.

$\endgroup$
5
  • 1
    $\begingroup$ I just fit a logistic regression to $5$ million observations in a few seconds, and all I did was run L <- glm(y ~ x1 + x2, family = binomial). What is the issue? $\endgroup$
    – Dave
    Jul 27 '21 at 13:51
  • $\begingroup$ Thanks for the comment. My problem was the random part (mixed logistic regression). $\endgroup$
    – Kirsten
    Jul 28 '21 at 7:34
  • 1
    $\begingroup$ Then the structure of the random effect is your issue. It's not just the size of the data, it is its structure. Your issue is not correctly defined. $\endgroup$
    – Roland
    Jul 29 '21 at 15:17
  • 1
    $\begingroup$ Do you have suggestions of how i can modernize the structure? or things that might guide me to find the problem? $\endgroup$
    – Kirsten
    Jul 30 '21 at 6:28
  • 1
    $\begingroup$ The structure of your data is the structure of your data. It is an inherent property that can't be changed. You can only account for it in your analyses and that needs to be tailored to your specific data. General advice could fill a few lectures. $\endgroup$
    – Roland
    Jul 30 '21 at 7:47
1
+50
$\begingroup$

As mentioned in the comments, 5 million is not a large hurdle unless you have a very large quantity of covariates. I'll answer this question from the perspective of applied economics, my field of interest.

For an econometric study on techniques in linear regression with an emphasis on causality, you may find useful any course that uses Angrist and Pishke Mostly Harmless Econometrics. This will skip a lot of the theory but still provide a robust understanding of what happens when you run a regression and how to interpret the results. The book is not technically difficult (maybe at the level of an intermediate undergrad stats course), though depending on your background it can be conceptually challenging.

There are many classes that use this book. Here is one from MIT that focuses on big data, if that is your interest. Some of the lecture notes are online. https://ocw.mit.edu/courses/economics/14-387-applied-econometrics-mostly-harmless-big-data-fall-2014/syllabus/

If you are working with time series, you will need a course that specializes in this. Two resources to consider. First another MIT OCW class: https://ocw.mit.edu/courses/economics/14-384-time-series-analysis-fall-2013/syllabus/ Alternatively, if you prefer self-study, consider working through Cochrane's time series notes, which I found intuitive and accessible: https://www.johnhcochrane.com/research-all/time-series-for-macroeconomics-and-finance

Hope this helps!

$\endgroup$
1
  • $\begingroup$ Thank you for assigning me the bounty- if you are satisfied with the answer, can you please accept the answer? Thank you! $\endgroup$
    – Matterhorn
    Sep 19 '21 at 4:23
1
$\begingroup$

For large data sets, once could use bootstrap aggregation to speed up and gain more accurate estimates. This article is a good exposition on how to do it: https://towardsdatascience.com/bootstrap-regression-in-r-98bfe4ff5007.

For glm, one could replace the lm function in the provided code and use whatever distribution family, whether Poisson, Binomial, etc.

For machine learning and other methods more ideal for big data, one can use Elements of Statistical Learning to study the theory: https://web.stanford.edu/~hastie/ElemStatLearn/.

I'm not familiar with machine learning for R or Stata beyond a few libraries.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.