Many statistical jobs ask for experience with large scale data. What are the sorts of statistical and computational skills that would be need for working with large data sets. For example, how about building regression models given a data set with 10 million samples?
$\begingroup$
$\endgroup$
3
asked Mar 2, 2011 at 19:05
-
1$\begingroup$ Some good pointers here. $\endgroup$– user22Commented Mar 3, 2011 at 16:03
-
$\begingroup$ It'd be helpful if you'd summarize the ones you think are best. $\endgroup$– rolando2Commented Mar 3, 2011 at 20:51
-
$\begingroup$ Also of interest is the related discussion of hypothesis testing with large sets of data: stats.stackexchange.com/q/2516/919 $\endgroup$– whuber ♦Commented Mar 7, 2011 at 16:00
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
7
Good answers have already appeared. I will therefore just share some thoughts based on personal experience: adapt the relevant ones to your own situation as needed.
For background and context--so you can account for any personal biases that might creep in to this message--much of my work has been in helping people make important decisions based on relatively small datasets. They are small because the data can be expensive to collect (10K dollars for the first sample of a groundwater monitoring well, for instance, or several thousand dollars for analyses of unusual chemicals). I'm used to getting as much as possible out of any data that are available, to exploring them to death, and to inventing new methods to analyze them if necessary. However, in the last few years I have been engaged to work on some fairly large databases, such as one of socioeconomic and engineering data covering the entire US at the Census block level (8.5 million records, 300 fields) and various large GIS databases (which nowadays can run from gigabytes to hundreds of gigabytes in size).
With very large datasets one's entire approach and mindset change. There are now too much data to analyze. Some of the immediate (and, in retrospect) obvious implications (with emphasis on regression modeling) include
Any analysis you think about doing can take a lot of time and computation. You will need to develop methods of subsampling and working on partial datasets so you can plan your workflow when computing with the entire dataset. (Subsampling can be complicated, because you need a representative subset of the data that is as rich as the entire dataset. And don't forget about cross-validating your models with the held-out data.)
Because of this, you will spend more time documenting what you do and scripting everything (so that it can be repeated).
As @dsimcha has just noted, good programming skills are useful. Actually, you don't need much in the way of experience with programming environments, but you need a willingness to program, the ability to recognize when programming will help (at just about every step, really) and a good understanding of basic elements of computer science, such as design of appropriate data structures and how to analyze computational complexity of algorithms. That's useful for knowing in advance whether code you plan to write will scale up to the full dataset.
Some datasets are large because they have many variables (thousands or tens of thousands, all of them different). Expect to spend a great deal of time just summarizing and understanding the data. A codebook or data dictionary, and other forms of metadata, become essential.
Much of your time is spent simply moving data around and reformatting them. You need skills with processing large databases and skills with summarizing and graphing large amounts of data. (Tufte's Small Multiple comes to the fore here.)
Some of your favorite software tools will fail. Forget spreadsheets, for instance. A lot of open source and academic software will just not be up to handling large datasets: the processing will take forever or the software will crash. Expect this and make sure you have multiple ways to accomplish your key tasks.
Almost any statistical test you run will be so powerful that it's almost sure to identify a "significant" effect. You have to focus much more on statistical importance, such as effect size, rather than significance.
Similarly, model selection is troublesome because almost any variable and any interaction you might contemplate is going to look significant. You have to focus more on the meaningfulness of the variables you choose to analyze.
There will be more than enough information to identify appropriate nonlinear transformations of the variables. Know how to do this.
You will have enough data to detect nonlinear relationships, changes in trends, nonstationarity, heteroscedasticity, etc.
You will never be finished. There are so much data you could study them forever. It's important, therefore, to establish your analytical objectives at the outset and constantly keep them in mind.
I'll end with a short anecdote which illustrates one unexpected difference between regression modeling with a large dataset compared to a smaller one. At the end of that project with the Census data, a regression model I had developed needed to be implemented in the client's computing system, which meant writing SQL code in a relational database. This is a routine step but the code generated by the database programmers involved thousands of lines of SQL. This made it almost impossible to guarantee it was bug free--although we could detect the bugs (it gave different results on test data), finding them was another matter. (All you need is one typographical error in a coefficient...) Part of the solution was to write a program that generated the SQL commands directly from the model estimates. This assured that what came out of the statistics package was exactly what went into the RDBMS. As a bonus, a few hours spent on writing this script replaced possibly several weeks of SQL coding and testing. This is a small part of what it means for the statistician to be able to communicate their results.
-
5$\begingroup$ +1, I'll share this wonderful response (and print it to have nearby ^_^) $\endgroup$ Commented Mar 7, 2011 at 15:29
-
2$\begingroup$ +1, this is what I'll certainly retell my students many years to come. $\endgroup$– mpiktasCommented Mar 8, 2011 at 13:40
-
2$\begingroup$ the anecdote reminded me the time when I had to transfer model from Eviews to R. The original model was done in Eviews, the result was about 20 equations. I had to present the results in the webpage with interactive interface. Since the model was work in progress, I wrote a code translating output of Eviews to R code with the same purpose that the exact model was used both in Eviews and in R. R worked very nicely, I even ended up using differencing the translated code for calculation of analytical gradient. $\endgroup$– mpiktasCommented Mar 8, 2011 at 14:29
-
3$\begingroup$ It is generally regarded as more constructive (if not simple courtesy) when downvotes are justified in a comment, unless there're obvious reasons not to do so (e.g., one-line vague response, no response to request for updating a wrong answer, offensive behavior). This contributes to enhance the quality of a response, when valid arguments are made. In this particular case, I see no reason for a downvote! $\endgroup$– chlCommented Dec 1, 2011 at 10:52
-
2$\begingroup$ +1 for automation reduces error: "write a program that generated the SQL commands directly from the model estimates". $\endgroup$– OrionCommented Jun 5, 2014 at 23:17
$\begingroup$
$\endgroup$
4
Your question should yield some good answers. Here are some starting points.
An ability to work with the tradeoffs between precision and the demands placed on computing power.
Facility with data mining techniques that can be used as preliminary screening tools before conducting regression. E.g., chaid, cart, or neural networks.
A deep understanding of the relationship between statistical significance and practical significance. A wide repertoire of methods for variable selection.
The instinct to crossvalidate.
-
$\begingroup$ I would also combine #4 and #1: its important to know how to cross validate without overwhelming your computing resources. $\endgroup$– ZachCommented Mar 3, 2011 at 19:39
-
1$\begingroup$ Could you explain your 2nd point? How would you use CHAID/CART/neural networks as screening tools for regression? $\endgroup$– raegtinCommented Mar 8, 2011 at 0:09
-
2$\begingroup$ @raegtin - I'm most familiar with CHAID, which comes up with so-called "interactions" that are often main effects masquerading as interactions because that is the only way the procedure will "let them in." (In CHAID there can be only 1 main effect identified as such, so all other main effects get squeezed into "interaction" cells.) But CHAID has the advantage of being able to check many many interactions. So once a few promising ones are identified, they can be incorporated into a regression or anova, with all of their lower-order components, and one can test for which ones are truly useful. $\endgroup$– rolando2Commented Apr 8, 2011 at 20:09
-
1$\begingroup$ +1 I am intrigued by the possibility of using data mining (especially CHAID) for exploring potential effects. It would be interesting to see an application, such as with the artificial (and small) dataset at stats.stackexchange.com/q/10363/919 $\endgroup$– whuber ♦Commented May 13, 2011 at 12:43
$\begingroup$
$\endgroup$
3
Good programming skills are a must. You need to be able to write efficient code that can deal with huge amounts of data without choking, and maybe be able to parallelize said code to get it to run in a reasonable amount of time.
-
4$\begingroup$ Coding is a must, but also knowing how to work with the OS not against it is important. You must understand that sometimes splitting the work has extra costs associated with it, as accessing disks and networks carries additional costs. You gotta understand different ways of blocking and waiting and doing interprocess communication. I've seen great scientific code that would spent most of it's time waiting for some system calls to finish. Befriend the sysadmin of your system, you can get a lot of help with optimization of yours systems by bringing them coffee ;) $\endgroup$– MarcinCommented Mar 7, 2011 at 13:28
-
2$\begingroup$ Sometimes it is better to write "Inefficient code" if this will aid in creating data structures which anticipate additional questions down the road which will probably be asked. $\endgroup$ Commented Mar 8, 2011 at 14:03
-
1$\begingroup$ @Ralph: +1, I absolutely agree and learned this the hard way myself. I didn't mean to apply that you should always write efficient code no matter what the tradeoffs, just that you should know how to. $\endgroup$– dsimchaCommented Mar 8, 2011 at 16:27
$\begingroup$
$\endgroup$
2
I would also add that the large scale data also introduces the problem of potential "Bad data". Not only missing data, but data errors and inconsistent definitions introduced by every piece of a system which ever touched the data. So, in additional to statistical skills, you need to become an expert data cleaner, unless someone else is doing it for you.
-Ralph Winters
-
4$\begingroup$ These are good points. Outliers and other data problems plague any dataset, no matter how large or small. In my experience they actually are easier to identify and deal with in large datasets, because you have the power to discriminate them from the mass of data and, especially if you use robust methods, they are less likely to influence the results. BTW, you're always doing "data cleaning" throughout any analysis. This isn't something that can be segregated and referred to a specialist to be handled once and for all. An outlier is only an outlier in the context of a particular model. $\endgroup$– whuber ♦Commented Mar 7, 2011 at 15:55
-
2$\begingroup$ Check out google refine as a semi automated data cleaner that helps avoid the pitfalls of hand editing. $\endgroup$ Commented Nov 29, 2011 at 2:48
$\begingroup$
$\endgroup$
1
- Framing the problem in the Map-reduce framework.
- The Engineering side of the problem, eg., how much does it hurt to use lower precision for the parameters, or model selection based not only on generalization but storage and computation costs as well.
answered Mar 3, 2011 at 2:00
-
$\begingroup$ +1 for mentioning about lower precision, though it is far from being an enginnering prerogative. The lower the precision the more likely we are to make bad decisions. This is closely tied to Type I/II error and spans several disciplines but is mostly relevant to statistics, decision science and economics. Utility functions should be thought of ahead of time and part of the thought process to identify a suitable methodology. $\endgroup$ Commented Jan 7, 2014 at 15:28