Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

My current project may require me to build a model to predict the behavior of a certain group of people. the training data set contains only 6 variables (id is only for identification purposes):

id, age, income, gender, job category, monthly spend

in which monthly spend is the response variable. But the training dataset contains approximately 3 million rows, and the dataset (which contains id, age, income, gender, job category but no response variable) to be predicted contains 1 million rows. My question is: is there any potential problems if I throw too many rows (3 million in this case) into a statistical model?I understand the computational expenses is one of the concern, are there any other concerns? Are there any books / papers that fully explain the data set size issue?

share|improve this question
14  
Having too many rows does not result in overfitting. Having too many columns does. –  Peter Flom Jan 27 at 18:21
4  
Because this question is predicated on multiple false assumptions--that people are reluctant to use many cases for analyses and that most statistical models cannot cope with large datasets--it is likely to accumulate irrelevant or confusing answers. Please consider editing your question to remove these misrepresentations. –  whuber Jan 27 at 19:00
    
edited, thanks for the correction –  user2926523 Jan 27 at 19:49
    
You still didn't take into account what @whuber said, though. The premises are wrong. It is not true that "most statistical models can't deal with huge datasets", so you're not going to get a useful answer to your question. It is not true either that you can use information on all people in a country at a given time (I guess this is what you mean by "population"). –  pkofod Jan 27 at 20:03
1  
ID is, if this is a sane analysis, not a single continuous variable. ID is a categorical variable because the differences between individuals does not correspond to the arbitrary numerical assignment they receive. That means you have 1 variable for each individual in the analysis. Potentially millions. –  AdamO Jan 27 at 20:27

2 Answers 2

There are two sorts of problems you might encounter:

1) Computer problems because the data set is too big. These days, a few million rows with 6 columns is just not that big. But, depending on your program, your computer, your amount of RAM and probably other things, it might bog down.

2) Statistical problems. Here, a problem like you discuss will have one "problem" that I know of: Even tiny effects will be highly significant. This is not really a problem with regression, it's a problem with p values. Better to look at effect sizes (regression parameters).

3) Another sort of problem with your model is not due to number of rows, but the nature of the response variable (monthly spend). Although OLS regression does not make any assumptions about the distribution of the response (only about the error), nevertheless, models with money as the dependent variable often have non-normal errors. In addition, it often makes sense, substantively, to take the log of the response. Whether this is so in your case depends on exactly what you are trying to do.

share|improve this answer

What's important is the number of individuals (rows) compared to the number of coefficients you need to estimate for the model you want to fit. Typical rules of thumb suggest about 20 observations per coefficient as a minimum, so you should be able to estimate up to 150,000 coefficients—surely more than adequate for your four predictors.

In fact you have an opportunity, not a problem, in this case: to fit a rather complex model including non-linear relationships of the response to predictors, & interactions between predictors; which may predict the response much better than a simpler one in which the relationships of the response to predictors are assumed to be linear & additive.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.