I'm currently studying deep learning using the textbook Deep Learning (Goodfellow, Bengio, & Courville - 2015) and had a question regarding a concept of machine learning provided in the book.

Specifically, the sentence I read is on page 115 under 5.2. Regularization and states as follows:

The no free lunch theorem implies that we must design our machine learning algorithms to perform well on a specific task.

I'm a bit confused by this statement. My understanding of the NFL theorem is that if we average each algorithm's performance over every existing sample, then their performances are pretty much the same.

If my understanding is correct, doesn't the above statement contradict the NFL theorem because it's implying that an algorithm performs better than another at a given task?

  • 1
    $\begingroup$ From tag excerpt for theory: For questions about statistical theory. Always include a more specific tag as well. Could you add one, given that machine-learning is not specific? $\endgroup$ Jun 19 '19 at 14:20
  • $\begingroup$ Sure, my bad. Thanks for the tip. $\endgroup$
    – Sean
    Jun 19 '19 at 14:23

For the NFL theorem, you're right. I think it doesn't contradict and they mean something like this: if you design your ML algorithm for your specific task, it'll succeed on it, and possibly fail on many others; which is in parallel to what the original paper by Wolpert says according to wiki: "any two optimization algorithms are equivalent when their performance is averaged across all possible problems". So, it basically states the importance of exploiting the properties of the particular problem of interest and not chasing for a generally better (in terms of possible problems) one.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.