Comparing classification results from models selected differently with Kruskal-Wallis For a seminar at university I'm reviewing a paper that proposes a new feature selection algorithm. The authors also evaluated their algorithm by applying it beside two other feature selection algorithms on 17 datasets. In order to compare them, they used the resulting feature subsets for classification and documented the accuracy and AUROC of the different algorithms.
In order to see if the presented algorithm is doing better than the other two, I'm separating their results into three groups (one for each algorithm) with each 17 instances (since they tested them for 17 datasets). Then I use the Kruskal-Wallis test to check if there is a significant difference in the distributions of the given quality measures for the algorithms.
My question is: is this approach valid? If not, how would you approach this task?
 A: Since the models are tested on the same datasets, they wouldn't be independent.  (Try correlating the AUCs from the different models, most likely the dataset with the highest value from one model will be the dataset with the highest value from another model.)  I might try Friedman's test instead of the KW.  
However, you won't have much power this way, which is especially problematic if you want to argue for the null hypothesis (which is already problematic, see my answer here: Why do statisticians say a non-significant result means “you can't reject the null” as opposed to accepting the null hypothesis?).  The reason is that the AUC has aggregated (thrown away) a lot of information about all of the instances in the dataset and their relative rankings into a single value.  You could get somewhat more information from the model outputs by using something like the Brier score instead of the AUC.  Depending on how much work you want to do (I suspect it would be a lot), you could try to work with all the observations from each dataset in a hierarchical model rather than aggregating at all.  
