Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a massive dataset, including about 5,000,000 points. There are 4 independent variables and two highly correlated dependent variables.

How should I do the regression analysis?

@StephanKolassa have told me to make a Cross-validation experiment and use the MAD as a measure to choose the best model from several alternatives. It is a very nice suggestion. But the problem is, how to get the "several alternative model" ? what methods or statistical software are recommended? Thank you!

My independent variable are the interplanetary condition components, and the dependent variable is the latitude of auroral oval boundary.

So far, the specific relationship is still unknown in physical principle, what we want to do is to get a model from the massive data which shows how these independent variables affect the dependent variable.

share|improve this question
I want to ask, as suggested by @Stephan, with the cross-validation method to decide which model best fits the data. But how to judge which model is the best among the alternative models? according to the regressors,R²,RMSE, or what? – yang1986 Nov 21 '12 at 12:16
Is this time series data? Is it valid to think that one regression model will fit all of the data or would it make more sense to perform regression on smaller blocks of points? I only ask because with the data I work with, this is the case. – Henry B. Nov 21 '12 at 13:03
NO,it is not time series data. I want to obtain one regression model that will fit all of the data. The problem is, how to judge the model ? – yang1986 Nov 21 '12 at 13:26
@yang1986: I edited my answer to suggest that you calculate the MAD between predictions and actuals in the holdout sample. If you do five-fold cross validation, you will end up with five MADs for each model you try (one from each holdout sample). Just average them, and you have one grand MAD per model. The model with the lowest MAD looks best. Alternatives would be Root Mean Squared Error etc. – Stephan Kolassa Nov 21 '12 at 15:24
Even if you include all interactions among the IVs, the regression analysis will only involve $16$-dimensional matrices (and their coefficients can be calculated in a fraction of a second using compiled code, and perhaps a minute on an interpreted platform like R: my machine takes 16 seconds for the simpler case with no interactions and that is primarily because R creates and saves a copy of the data). If you randomly select, say, a few thousand observations for regression, that might be good enough to tell you everything you need to know before you spend time computing with all the data. – whuber Nov 21 '12 at 15:24

The main thing to keep in mind is that with this amount of data, every coefficient will probably come out as statistically significant.

In order to find out which regressors are really important (as contrasted with statistically significant), I recommend using a holdout sample: fit your model to only 4 million data points, predict the other million points and compare to the actual values. Do this for a couple of different models (using or not using regressors, transforming regressors etc.) and see which ones yield the best predictions, by e.g. calculating the Mean Absolute Deviation (MAD) between the predictions and the actual observations.

Better yet: iterate this over the entire dataset five times, using a different million points as a holdout sample each time. This is known as "cross-validation" (five-fold cross-validation in this case).

share|improve this answer
+1 for the point that computing power is not the challenge here - just the problems with translating statistical inference to large datasets. (I also gave +1 to @mpiktas; it only took around 40 seconds to run a simulated example of this size on my far-from-cutting-edge laptop). – Peter Ellis Nov 21 '12 at 10:24
Thanks a million for your good suggestion! – yang1986 Nov 21 '12 at 11:37
The effect of a variable is in no way guaranteed to be significant just because you have many observation. – Repmat Dec 4 '15 at 22:35

You have 6 variables and 5 milion data points. So your data set would take about half a gigabyte of memory ($\frac{5\cdot 10^6\cdot16}{1024^2}\cdot 6$). So it is not that big for computers which now usually have 4GB RAM as a standard. The point I am trying to make is that although your data is big it is not massive and so you can do usual regression analysis.

share|improve this answer
could you write your formula in LaTeX? Since you have only exterior parens, right now it looks like you mean $\frac{5*10^6*16}{1024^2*6}$ which would be $\frac{80*10^6}{6,291,456}$ but I am not sure if that is what you meant. – Peter Flom Nov 21 '12 at 11:14
Fixed it, original was copied from my R console. – mpiktas Nov 21 '12 at 11:23
I see, thank you very much! – yang1986 Nov 21 '12 at 11:36
Haha, I read the question as $6\cdot 5\times10^9$, and I read your answer, and was like "that can't be right, I've gotta test that", so I tried a <- runif(2000000000) in R. Used up 6 gig of ram, and another 9 gig of swap. Was going to make some snotty comment, until I re-read the question. Luckily, I didn't need my computer for the last half hour :D – naught101 Nov 21 '12 at 11:55
Though I have had problems with datasets with $25\cdot 1.2\times 10^6$ in 32 bit R before. Can't remember exactly what the cause was, and no longer have such a machine to test it on... – naught101 Nov 21 '12 at 11:57

take a look at LASSO estimators - those were developed for large data

share|improve this answer
Can you please elaborate? Besides, I doubt lasso estimators were specifically developed for long data; arguably they can shed some light on wide data with many variables, but that does not seem to be the OPs problem. – StasK Dec 4 '15 at 21:14
This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post - you can always comment on your own posts, and once you have sufficient reputation you will be able to comment on any post. - From Review – John Dec 4 '15 at 21:26

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.