Modification of "corrected repeated k-fold cv test” when also averaging Random Forest results across multiple sampling seeds? I would be very grateful for any ideas concerning the following problem. I would be even more grateful if someone could point me to a literature reference proposing a solution for a scenario like the one I describe below. In short, I am looking to adapt Bouackert and Frank's 2004 article on corrected repeated k-fold cv test (PDF version) for a scenario in which the classifier is also intrinsically random.
I am a student in cheminformatics (i.e. would appreciate mathematical jargon kept to a minimum if possible).
The scenario:

*

*I am comparing two descriptor/attribute/feature sets (for the purpose of building classifiers using Machine Learning - the classes might be toxic vs. nontoxic). For example - encoding molecules (instances) in my dataset by a vector of numbers corresponding to a) a 2D vs. b) a 3D representation of molecular structure.


*In both cases, I generate k-fold cross-validation results (non-overlapping folds) using Random Forest acros R repetitions of cross-validation. Furthermore, for each train/validation set pair (corresponding to a given value of R and k), I repeat model generation and validation Q times - each time, using a different seed to initialise the random number generator for Random Forest.
E.g. for a given value of k and R (I am working in the R programming language):
    set.seed(CURRENT_SEED_VALUE) 
    rf <- randomForest(current_train_x,current_train_y)
    predictions <- predict(rf,current_test_x)
    figure_of_merit <- fom_function(predictions,current_test_y)



*I can then take the arithmetic mean of all figure_of_merit values (for both sets of descriptors). I wish to get a p-value corresponding to the null-hypothesis: The mean difference in this figure of merit for both methods is zero.

Supposing I was not considering multiple randomForest RNG seeds, I could obtain such a p-value using Bouckaert and Frank's 2004 “corrected repeated k-fold cv test”.
But, how could I modify this to get a p-value for the overall mean difference???
(The only approach I can currently think of is getting one p-value per RNG seed value, then (after correcting for multiple testing using p.adjust(...)) seeing whether the p-value is statistically significant for all RNG seeds.)
 A: I am also a chemoinformatician, so I can be wrong, but since there is no any answer, I'll put here my suggestions. Really, I don't see a problem here. The mean of random variables is also a random variable. So, you have to obtain a k x R matrix of *mean_figure_of_merit2D* (each mean is taken from Q *figure_of_merit2D*). Then transform this matrix to a vector like
dim(mean_figure_of_merit2D)<-c(1,k*R)
(You can also go straightly to the vector form in the cross-validation cycle)
Obtain *mean_figure_of_merit3D* the same way. And then - as the Bouckaert and Frank's paper stating -
x<-mean_figure_of_merit2D - mean_figure_of_merit3D
m<-mean(x)
s<-sd(x)
t=m/sqrt((1/(k*R)+n2/n1)*s^2
More repetitions (R) are made - then the distribution of t will be closer to Student's t-distribution - more correct p-value will be obtained.
BUT even if you'll prove that 3D descriptors are better than 2D (or vice versa) - this inference has no value, since it concerns the prespecified dataset treated with prespecified algorithm that was run with the prespecified parameters - and thus cannot be extrapolated. I'll better recommend you to combine your descriptors together and thus increase the chance to obtain more predictive model. And one else point: since random forest algorithm is using bagging, you don't need to cross-validate your model - the out-of-bag statistics is unbiased (see http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#ooberr). 
