# Modification of "corrected repeated k-fold cv test” when also averaging Random Forest results across multiple sampling seeds?

I would be very grateful for any ideas concerning the following problem. I would be even more grateful if someone could point me to a literature reference proposing a solution for a scenario like the one I describe below. In short, I am looking to adapt Bouackert and Frank's 2004 article on corrected repeated k-fold cv test (PDF version) for a scenario in which the classifier is also intrinsically random.

I am a student in cheminformatics (i.e. would appreciate mathematical jargon kept to a minimum if possible).

The scenario:

1. I am comparing two descriptor/attribute/feature sets (for the purpose of building classifiers using Machine Learning - the classes might be toxic vs. nontoxic). For example - encoding molecules (instances) in my dataset by a vector of numbers corresponding to a) a 2D vs. b) a 3D representation of molecular structure.

2. In both cases, I generate k-fold cross-validation results (non-overlapping folds) using Random Forest acros R repetitions of cross-validation. Furthermore, for each train/validation set pair (corresponding to a given value of R and k), I repeat model generation and validation Q times - each time, using a different seed to initialise the random number generator for Random Forest.

E.g. for a given value of k and R (I am working in the R programming language):

set.seed(CURRENT_SEED_VALUE)
rf <- randomForest(current_train_x,current_train_y)
predictions <- predict(rf,current_test_x)
figure_of_merit <- fom_function(predictions,current_test_y)

3. I can then take the arithmetic mean of all figure_of_merit values (for both sets of descriptors). I wish to get a p-value corresponding to the null-hypothesis: The mean difference in this figure of merit for both methods is zero.

Supposing I was not considering multiple randomForest RNG seeds, I could obtain such a p-value using Bouckaert and Frank's 2004 “corrected repeated k-fold cv test”.

But, how could I modify this to get a p-value for the overall mean difference???

(The only approach I can currently think of is getting one p-value per RNG seed value, then (after correcting for multiple testing using p.adjust(...)) seeing whether the p-value is statistically significant for all RNG seeds.)

-
I am also a chemoinformatician, so I can be wrong, but since there is no any answer, I'll put here my suggestions. Really, I don't see a problem here. The mean of random variables is also a random variable. So, you have to obtain a k x R matrix of *mean_figure_of_merit2D* (each mean is taken from Q *figure_of_merit2D*). Then transform this matrix to a vector like
dim(mean_figure_of_merit2D)<-c(1,k*R)
x<-mean_figure_of_merit2D - mean_figure_of_merit3D m<-mean(x) s<-sd(x) t=m/sqrt((1/(k*R)+n2/n1)*s^2