Material/courses for analyzing very large data sets

Question

I am looking for good materials to learn more about analyzing very large data sets, say 5 million observations, with a reasonable computational time (say less than 10h).

My main interests is traditional analysis methods, such as linear, Poisson and logistic regression, both mixed and not-mixed. Together with simple illustration and presentation of data, that provide easy understandable interpretation and options for confounder correction. I am open to alternative methods yielding reasonable similar results with the methods mentioned above, but reducing the computational time. For example, in some cases using a robust sandwich estimator can account for incorrect specified correlation structure but reduce computational time compared with a cluster methods.

I am familiar with the programs R, SAS and STATA, and would prefer methods available in these programs.

If you know of relevant (English/Danish) courses held by universities, I would love to have links for course descriptions for this as well.

I hope to hear from you.

Answer 1

As mentioned in the comments, 5 million is not a large hurdle unless you have a very large quantity of covariates. I'll answer this question from the perspective of applied economics, my field of interest.

For an econometric study on techniques in linear regression with an emphasis on causality, you may find useful any course that uses Angrist and Pishke Mostly Harmless Econometrics. This will skip a lot of the theory but still provide a robust understanding of what happens when you run a regression and how to interpret the results. The book is not technically difficult (maybe at the level of an intermediate undergrad stats course), though depending on your background it can be conceptually challenging.

There are many classes that use this book. Here is one from MIT that focuses on big data, if that is your interest. Some of the lecture notes are online. https://ocw.mit.edu/courses/economics/14-387-applied-econometrics-mostly-harmless-big-data-fall-2014/syllabus/

If you are working with time series, you will need a course that specializes in this. Two resources to consider. First another MIT OCW class: https://ocw.mit.edu/courses/economics/14-384-time-series-analysis-fall-2013/syllabus/ Alternatively, if you prefer self-study, consider working through Cochrane's time series notes, which I found intuitive and accessible: https://www.johnhcochrane.com/research-all/time-series-for-macroeconomics-and-finance

Hope this helps!

Answer 2

For large data sets, once could use bootstrap aggregation to speed up and gain more accurate estimates. This article is a good exposition on how to do it: https://towardsdatascience.com/bootstrap-regression-in-r-98bfe4ff5007.

For glm, one could replace the lm function in the provided code and use whatever distribution family, whether Poisson, Binomial, etc.

For machine learning and other methods more ideal for big data, one can use Elements of Statistical Learning to study the theory: https://web.stanford.edu/~hastie/ElemStatLearn/.

I'm not familiar with machine learning for R or Stata beyond a few libraries.