is permutation testing functionally equivalent to training/test? If permutation testing is applied in machine learning, is permutation testing functionally equivalent to training/test?
 A: No, they are different and used for different purposes.


*

*Train/test splits are used to measure generalization predictive performance: the performance of predictions done on unknown data.

*Permutation tests establish a sort of baseline: you simulate a no-information situation and then check how your figure of merit behaves (is distributed) for this no-information data. 


Purely invented example: you could use train/test splits to establish that your fancy new classifier has 85 % sensitivity. You can use a permutation test to establish that a naive purely guessing classifier on your data has an expected sensitivity of 75 %. You can also look at the distribution of observed sensitivities during the permutation test and find that 15 % of the permutations had observed sensitivity > 90 %, so forcing you to conclude that 85 % sensitivity may not be that much of an achievement in that particular situation.
Actually, the two techniques are orthogonal in the sense that you can combine them: you can e.g. do a permutation test on your train/test splitting techique. If you then see better than random guessing performance, you have indication of certain types of data leaks. (Note though, that if you see performance like random guessing other types of data leaks are still possible. The permutation can indicate something is wrong, but it cannot prove that everything is right.) 
Permutation tests and train/test splits are related in that they are both tools that can be used during validation (which in turn comprises far more things one can do besides those two techniques). 
