Does "Brute Force Optimization" violate the "No Free Lunch Theorem"? The No Free Lunch Theorem says that no learning algorithm is better than another. What about brute force? Brute force works on all problem cases.
 A: Brute force is a very very vague term and is not on the same level of hierarchy as individual learning algorithms. 
Rather it is one potential strategy for using a range of individual learning algorithms, one that can be very resource intensive and so is not a universal panacea. Which proves 'no free lunch' applies to brute force. You pay for maximising the chances of building a model in physical money, equipment and personnel.
A: The seemingly novel implication for the No Free Lunch theorem arises as a direct result of the equally novel way in which the problem is posed here. As St. Thomas Aquinas once said, “An error in the beginning is an error indeed.” The logical implications are consistent, but only after the introduction of a flawed premise.
• "The No Free Lunch Theorem says that no learning algorithm is better than another." The premise of the question is not precisely true; the theorem actually says that no single algorithm will perform better than their competitors on all datasets, not that all algorithms are created equal. One could easily design algorithms specifically to avoid convergence, so that they never hit certain targets. Trivial examples would include simply changing the inner workings of a maximization algorithm so that they instead minimize the criterion, or vice-versa. The set of such lousy algorithms guaranteed to never converge is practically infinite.
• Yes, "brute force works on all problem classes," in the sense that nothing's stopping us from machining our data in this way, in the hopes of getting the correct output through a sheer lucky break (i.e. random chance). But that does not mean it does so more efficiently than other algorithms for all datasets. Only if that were true would it violate the No Free Lunch Theorem.
A brute force algorithm might actually perform as well as any SVM, neural net or sophisticated regression method for problems where the solution surface is shaped like a golf course with a single hole, so that wrong answers provide zero information about where the single solution might reside. Only enumerating all possible solutions would solve this problem, which is akin to perfect encryption. For almost all other problems, other machine learning algorithms will be able to find the solution several orders of magnitude faster than simply enumerating all solutions, as a brute force approach does.
• Brute force won't necessarily converge if the problem definition and/or dataset changes during computation; the new convergence point might be within the set of solutions it has already tried, rejected and will never revisit. If the algorithm is designed to reevaluate old solutions, there is no guarantee that the algorithm will ever pinpoint a moving target of this kind before it changes again. Convergence for brute force is only guaranteed for static problems. Even then, it is of often little practical value, since calculating many problems of only moderate complexity with sheer brute force would take many times the lifetime of the universe, even with improved computer technology.
