# Is it true that a larger, representative dataset is always better to use than a smaller, representative dataset?

By "representative" I mean that the data in the dataset faithfully reflects the "underlying signal" a model is trying to tap in to. Is it always true that, as long as increasing the size of the dataset has not added bias, ie it is still just as representative of the underlying signal as the smaller dataset, that the more data the merrier for the model?

What does it mean if a dataset has a higher accuracy on a smaller dataset? Is that a sign that it's overfitting on the smaller dataset, not that it's a better idea to use the smaller dataset? Should it be expected to do better or worse than a larger dataset?

• You can always subsample a larger representative dataset, so perforce the larger one cannot be any worse for any purpose, unless subsampling turns out to be a difficult operation. Overfitting is not a property of a sample: it's a property of the procedure you use to analyze the sample.
– whuber
Mar 21, 2022 at 15:34
• Just to be clear, representative means the same thing for both datasets. Presumably, datasets randomly sampled, etc. Mar 21, 2022 at 16:41
• Just to be sure, by large or small do you mean number of data points or number of variables? Mar 21, 2022 at 18:58
• @RichardHardy Number of data points. Mar 21, 2022 at 18:59

• @Sangstar If you're asking about precision, then assuming the data are a simple random sample and do not impart bias on the resulting estimate, then more data is always better for precision since the standard error is inversely proportional to $\sqrt{n}$. Unfortunately, most claims in statistics can not be so general. You have to provide very specific circumstances, as I have done specifying a simple random sample. Mar 21, 2022 at 15:44