As far as I know, much of the statistical and machine learning literature where modeling algorithms are compared for their generalization error performance as a function of sample size (think of Andrew Ng's famous graph of model performance as a function of size comparing "classical" statistical methods to neural networks) suffers from a critical error. Either training samples of varying size are taken from the exact same dataset or samples of varying size across varying study domains are used to make a claim about the performance of one algorithm or another.

Yet when we compare model performance by sample size using samples of varying size from the same dataset, we assume that the quality of data is constant across sample sizes. When we compare model performance across samples of varying size that come from separate study domains, our assessment of model performance as a function of sample size is confounded potentially by the signal-to-noise ratio of the study domain.

This is problematic because data quality is most likely dependent on sample size. For example, large samples mined from the web or a company's data warehouse have unknown biases that, in my experience, few analysts adequately account for, especially analysts trained in programs that focus heavily on data mining, model-fitting, and cross-validation techniques at the expense of sampling and measurement theory.

So are there any studies that address these issues in their comparison of various algorithms of varying computational sophistication and complexity? Am I mischaracterizing the literature when I say much of it is flawed in the way I've described?


I think there are different perspectives to this problem:

  • Here's a paper raising concerns about data quality that are somewhat related to yours in the literature: Berget & Næs: Using unclassified observations for improving classifiers, J Chemom, 2004, 18, 103-111
    They look at situations where semi-supervised models are proposed and find that data quality for cases where reference is available is often better than for larger unlabeled data sets and state:

    Frequently the unclassified objects are collected under less controlled conditions than the observations used for training. The accuracy of routine measurements may, for instance, be much poorer than when collecting data for the training set. Hence the unclassified data may contain erroneous observations or outliers.

  • Also, it is certainly good to spell out the assumption that data quality is equal across the whole sample.

  • On the other hand, this assumption implicitly covered by the assumption that there is such a thing as one population with some distribution that is sampled and then modeled. And that is a very important and basic assumption: without, the whole approach often falls apart.

  • And yes, the situation you describe may also be descibed as having a sample that is obtained by pooling a number of rather distinct samples. So this description may lead to another set of search terms for your question.

  • Another search term that I'd check in my field (chemometrics/analytical chemistry) is ruggedness. Ruggedness describes how well a method is able to deal with (slightly) altered contditions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.