I would be very grateful for any ideas concerning the following problem. I would be even more grateful if someone could point me to a literature reference proposing a solution for a scenario like the one I describe below. In short, I am looking to adapt Bouackert and Frank's 2004 article on corrected repeated k-fold cv test (PDF version) for a scenario in which the classifier is also intrinsically random.
I am a student in cheminformatics (i.e. would appreciate mathematical jargon kept to a minimum if possible).
I am comparing two descriptor/attribute/feature sets (for the purpose of building classifiers using Machine Learning - the classes might be toxic vs. nontoxic). For example - encoding molecules (instances) in my dataset by a vector of numbers corresponding to a) a 2D vs. b) a 3D representation of molecular structure.
In both cases, I generate k-fold cross-validation results (non-overlapping folds) using Random Forest acros R repetitions of cross-validation. Furthermore, for each train/validation set pair (corresponding to a given value of R and k), I repeat model generation and validation Q times - each time, using a different seed to initialise the random number generator for Random Forest.
E.g. for a given value of k and R (I am working in the R programming language):
set.seed(CURRENT_SEED_VALUE) rf <- randomForest(current_train_x,current_train_y) predictions <- predict(rf,current_test_x) figure_of_merit <- fom_function(predictions,current_test_y)
I can then take the arithmetic mean of all figure_of_merit values (for both sets of descriptors). I wish to get a p-value corresponding to the null-hypothesis: The mean difference in this figure of merit for both methods is zero.
Supposing I was not considering multiple randomForest RNG seeds, I could obtain such a p-value using Bouckaert and Frank's 2004 “corrected repeated k-fold cv test”.
But, how could I modify this to get a p-value for the overall mean difference???
(The only approach I can currently think of is getting one p-value per RNG seed value, then (after correcting for multiple testing using
p.adjust(...)) seeing whether the p-value is statistically significant for all RNG seeds.)