9
$\begingroup$

My current project may require me to build a model to predict the behavior of a certain group of people. the training data set contains only 6 variables (id is only for identification purposes):

id, age, income, gender, job category, monthly spend

in which monthly spend is the response variable. But the training dataset contains approximately 3 million rows, and the dataset (which contains id, age, income, gender, job category but no response variable) to be predicted contains 1 million rows. My question is: is there any potential problems if I throw too many rows (3 million in this case) into a statistical model?I understand the computational expenses is one of the concern, are there any other concerns? Are there any books / papers that fully explain the data set size issue?

Improve this question
$\endgroup$
  • 15
    $\begingroup$ Having too many rows does not result in overfitting. Having too many columns does. $\endgroup$ – Peter Flom Jan 27 '14 at 18:21
  • 5
    $\begingroup$ Because this question is predicated on multiple false assumptions--that people are reluctant to use many cases for analyses and that most statistical models cannot cope with large datasets--it is likely to accumulate irrelevant or confusing answers. Please consider editing your question to remove these misrepresentations. $\endgroup$ – whuber♦ Jan 27 '14 at 19:00
  • $\begingroup$ You still didn't take into account what @whuber said, though. The premises are wrong. It is not true that "most statistical models can't deal with huge datasets", so you're not going to get a useful answer to your question. It is not true either that you can use information on all people in a country at a given time (I guess this is what you mean by "population"). $\endgroup$ – pkofod Jan 27 '14 at 20:03
  • 2
    $\begingroup$ ID is, if this is a sane analysis, not a single continuous variable. ID is a categorical variable because the differences between individuals does not correspond to the arbitrary numerical assignment they receive. That means you have 1 variable for each individual in the analysis. Potentially millions. $\endgroup$ – AdamO Jan 27 '14 at 20:27
  • $\begingroup$ thanks for the comment, but i still get confused why i cant use information on all people in a country at a given time (this is a contrived example, so lets assume i have that info)? $\endgroup$ – user2926523 Jan 27 '14 at 20:39
12
$\begingroup$

There are two sorts of problems you might encounter:

1) Computer problems because the data set is too big. These days, a few million rows with 6 columns is just not that big. But, depending on your program, your computer, your amount of RAM and probably other things, it might bog down.

2) Statistical problems. Here, a problem like you discuss will have one "problem" that I know of: Even tiny effects will be highly significant. This is not really a problem with regression, it's a problem with p values. Better to look at effect sizes (regression parameters).

3) Another sort of problem with your model is not due to number of rows, but the nature of the response variable (monthly spend). Although OLS regression does not make any assumptions about the distribution of the response (only about the error), nevertheless, models with money as the dependent variable often have non-normal errors. In addition, it often makes sense, substantively, to take the log of the response. Whether this is so in your case depends on exactly what you are trying to do.

Improve this answer
$\endgroup$
  • $\begingroup$ Hi Peter, can you please explain in more detail why the response variable (i.e., monthly spend) should be taken log? What benefit can we get from this? $\endgroup$ – shihpeng Jun 13 '16 at 17:52
  • 1
    $\begingroup$ Sure. Monetary values should often be logged. We think of money in multiplicative, not additive terms. If you go from spending $1000 to $1010 that's nothing. If you go from $10 to $20 that's huge. $\endgroup$ – Peter Flom Jun 15 '16 at 21:15
2
$\begingroup$

What's important is the number of individuals (rows) compared to the number of coefficients you need to estimate for the model you want to fit. Typical rules of thumb suggest about 20 observations per coefficient as a minimum, so you should be able to estimate up to 150,000 coefficients—surely more than adequate for your four predictors.

In fact you have an opportunity, not a problem, in this case: to fit a rather complex model including non-linear relationships of the response to predictors, & interactions between predictors; which may predict the response much better than a simpler one in which the relationships of the response to predictors are assumed to be linear & additive.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.