Does the "No Free Lunch Theorem" apply to general statistical tests? A woman I was working for asked me to do a one-way ANOVA on some data. I replied that the data were repeated measures (time series) data, and that I thought the assumption of independence was violated. She replied that I should not worry about the assumptions, just do the test and she would take into account that the assumptions might not have been met.
That did not seem right to me. I did some research, and found this wonderful blog post by David Robinson, K-means clustering is not a free lunch, which exposed me to the No Free Lunch Theorem. I have looked at the original paper and some follow on stuff, and frankly the math is a bit over my head.
The gist of it -- according to David Robinson -- seems to be that the power of a statistical test comes from its assumptions. And he gives two great examples. As I wade through the other articles and blog posts about it, it seems to always be referenced in terms of either supervised learning or search.
So my question is, does this theorem apply to statistical tests in general? In other words, can one say that the power of a t-test or ANOVA comes from its adherence to the assumptions, and cite the No Free Lunch Theorem? 
I owe my former boss a final document regarding the work I did, and I would like to know if I can reference the No Free Lunch Theorem in stating that you cannot just ignore the assumptions of a statistical test and say you'll take that into account when evaluating the results. 
 A: You can cite the No Free Lunch Theorem if you want, but you could also just cite the Modus Ponens (also known as the Law of Detachment, the basis of deductive reasoning), which is the root of the No Free Lunch Theorem.
The No Free Lunch Theorem encompass a more specific idea: the fact that there's no algorithm that can fit all purposes. In other words, the No Free Lunch Theorem is basically saying that there's no algorithmic magic bullet. This roots on the Modus Ponens, because for an algorithm or a statistical test to give the correct result, you need to satisfy the premisses.
Just like in all mathematical theorems, if you violate the premisses, then the statistical test is just empty of sense, and you cannot derive any truth from it. So if you want to explain your data using your test, you must assume that the required premisses are met, if they're not (and you know that), then your test is dead wrong.
That's because scientific reasoning is based on deduction: basically, your test/law/theorem is an implication rule, which says that if you have the premisse A then you can conclude B: A=>B, but if you don't have A, then you can either have B or not B, and both cases are true, that's one of the basic tenets of logical inference/deduction (the Modus Ponens rule). In other words, if you violate the premisses, the result doesn't matter, and you cannot deduce anything.
Remember the binary table of implication:
A   B   A=>B
F   F    T
F   T    T
T   F    F
T   T    T

So in your case, to simplify, you have Dependent_Variables => ANOVA_correct. Now, if you use independent variables, thus Dependent_Variables is False, then the implication will be true, since the Dependent_Variables assumption is violated.
Of course this is simplistic, and in practice your ANOVA test may still return useful results because there is almost always some degree of independence between dependent variables, but this gives you the idea why you just can't rely on the test without fulfilling the assumptions.
However, you can also use tests which premisses are not satisfied by the original by reducing your problem: by explicitly relaxing the independency constraint, your result may still be meaningful, althrough not guaranteed (because then your results apply to the reduced problem, not the full problem, so you cannot translate every results except if you can prove that the additional constraints of the new problem do not impact your test and thus your results).
In practice, this is often used to model practical data, by using Naive Bayes for example, by modelling dependent (instead of independent) variables using a model that assume independent variables, and surprisingly it works often very well, and sometimes better than models accounting for dependencies. You can also be interested by this question about how to use ANOVA when the data doesn't exactly meet all expectations.
To summary: if you intend to work on practical data and your goal is not to prove any scientific result but to make a system that just works (ie, a web service or whatever practical application), the independency assumption (and maybe other assumptions) can be relaxed, but if you're trying to deduce/prove some general truth, then you should always use tests which you can mathematically guarantee (or at least safely and provably assume) that you satisfy all premisses.
A: I don't know of a proof but I'll bet this applies quite generally.  An example is an experiment with 2 subjects in each of 2 treatment groups.  The Wilcoxon test cannot possibly be significant at the 0.05 level, but the t-test can.  You could say that its power comes more than half from its assumptions and not just from the data.  To your original problem, it is not appropriate to proceed as if the observations per subject are independent.  To take things into account after the fact is certainly not good statistical practice except in very special circumstances (e.g., cluster sandwich estimators).
