1
$\begingroup$

By "representative" I mean that the data in the dataset faithfully reflects the "underlying signal" a model is trying to tap in to. Is it always true that, as long as increasing the size of the dataset has not added bias, ie it is still just as representative of the underlying signal as the smaller dataset, that the more data the merrier for the model?

What does it mean if a dataset has a higher accuracy on a smaller dataset? Is that a sign that it's overfitting on the smaller dataset, not that it's a better idea to use the smaller dataset? Should it be expected to do better or worse than a larger dataset?

$\endgroup$
4
  • 2
    $\begingroup$ You can always subsample a larger representative dataset, so perforce the larger one cannot be any worse for any purpose, unless subsampling turns out to be a difficult operation. Overfitting is not a property of a sample: it's a property of the procedure you use to analyze the sample. $\endgroup$
    – whuber
    Mar 21, 2022 at 15:34
  • $\begingroup$ Just to be clear, representative means the same thing for both datasets. Presumably, datasets randomly sampled, etc. $\endgroup$
    – BruceET
    Mar 21, 2022 at 16:41
  • $\begingroup$ Just to be sure, by large or small do you mean number of data points or number of variables? $\endgroup$ Mar 21, 2022 at 18:58
  • $\begingroup$ @RichardHardy Number of data points. $\endgroup$
    – sangstar
    Mar 21, 2022 at 18:59

1 Answer 1

2
$\begingroup$

I'll interpret "representative" to mean "obtained without bias" such as selection bias. The question is then "is more data better than less?".

Assuming the data are a simple random sample from the appropriate population, then more data is usually better in so far as power, precision, etc are concerned. From more data, we are better equipped to model the conditional mean since we can spend more degrees of freedom (say, expanding the effect of some covariate using restricted cubic splines). However, there are non-statistical considerations we should highlight.

Randomized control trials (RCTs) often intend to use "enough" data. Too little, and the RCT may be under powered, but too much would be a waste of resources. Closely related to this consideration is the relative need for the additional precision; does matter if I can estimate the uncertainty in the effect to 8 decimal places? Its probably more important in say nano-technology than it would be in social science.

$\endgroup$
4
  • $\begingroup$ Is your usage of the word "usually" better in "usualy better in so far as power, precision, etc are concerned." to say that it's pretty much as a rule of thumb always better, except for special cases like RCTs? $\endgroup$
    – sangstar
    Mar 21, 2022 at 15:34
  • 1
    $\begingroup$ @sangstar I won't commit to a definitive answer unless we are talking about one very specific property. I'm hedging by bets lest someone comes with a corner case I haven't considered. $\endgroup$ Mar 21, 2022 at 15:41
  • $\begingroup$ One specific property? Which one would that be? Precision? Isn't it fair to say that in the majority of cases, more data is better for model accuracy on new data? $\endgroup$
    – sangstar
    Mar 21, 2022 at 15:42
  • $\begingroup$ @Sangstar If you're asking about precision, then assuming the data are a simple random sample and do not impart bias on the resulting estimate, then more data is always better for precision since the standard error is inversely proportional to $\sqrt{n}$. Unfortunately, most claims in statistics can not be so general. You have to provide very specific circumstances, as I have done specifying a simple random sample. $\endgroup$ Mar 21, 2022 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.