James et al. "An Introduction to Statistical Learning" (2013) p. 29 states:

There is no free lunch in statistics: no one method dominates all others over all possible data sets. On a particular data set, one specific method may work best, but some other method may work better on a similar but different data set.

The focus on data sets leaves an ambiguity as to whether the no free lunch phenomenon is due to
(1) random sampling from any given data generating process (DGP) or population, resulting in samples with different properties/patterns or
(2) the variety of possible DGPs and populations that are being modelled.
My understanding is that the no free lunch theorem concerns (2) rather than (1).

Quesion: Would it not be more precise and accurate to replace "data sets" with "data generating processes" in the passage?

  • $\begingroup$ Very interesting question. It's tempting to consider whether a rigorous answer can be formulated in terms of statistical decision theory. $\endgroup$ – Michael Sep 24 at 2:02

I think that the expression data set is used in a more broad sense than a single sample from a distribution. In practice, the expression data set is frequently applied to a collection of samples from the same population, such as a set measurements performed on different days and different subjects, all stitched together into a single object. If this is the case, the underlying statistical distribution is kind of in one-to-one correspondence to the dataset, because in practice nobody really will ever perform exactly the same experiment you did on exactly the same individuals. Under this terminology, the expressions data set and population are to some extent interchangeable. While this use of language may not be very sound mathematically, it is to some extent justified by the desire to simplify presentation for the target audience

However, under the assumption of the terminology you use, you are correct, IMHO. Clearly, if a method is adequately cross-validated on sufficient samples from a given population, the probability that it would perform differently on a different sample should be quite small.

| cite | improve this answer | |
  • $\begingroup$ Thank you for your answer. A method can certainly perform with varying success on different samples from the same population, especially if the samples are small and/or the method tends to overfit. However, in my understanding this is not what is meant by the no free lunch statement. I am also suspicious of equating data set and population. If that were the case, statistical and ML methods would be redundant besides the function of summarization, as there are no unseen data points to predict and all relationships are fully observable, thus no inference needed; I doubt this is usually so. $\endgroup$ – Richard Hardy Sep 23 at 15:50
  • $\begingroup$ The bigger sense of dataset in my experience is just meaning all the data being analysed, with minimally several observations and likely several variables. I guess this answer is using sample in the scientific rather than the statistical sense. $\endgroup$ – Nick Cox Sep 23 at 15:53
  • $\begingroup$ @NickCox, your take on my question would also be appreciated. $\endgroup$ – Richard Hardy Sep 23 at 16:01
  • $\begingroup$ I am aware of theorems with this name but I am inclined to interpret this as a variant on the answer depending on the question and the circumstances. $\endgroup$ – Nick Cox Sep 23 at 16:04
  • 1
    $\begingroup$ Thank you for a helpful discussion. $\endgroup$ – Richard Hardy Sep 30 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.