I would be very grateful for any ideas concerning the following problem. I would be even more grateful if someone could point me to a literature reference proposing a solution for a scenario like the one I describe below. In short, I am looking to adapt Bouackert and Frank's 2004 article on corrected repeated k-fold cv test (PDF version) for a scenario in which the classifier is also intrinsically random.

I am a student in cheminformatics (i.e. would appreciate mathematical jargon kept to a minimum if possible).

The scenario:

  1. I am comparing two descriptor/attribute/feature sets (for the purpose of building classifiers using Machine Learning - the classes might be toxic vs. nontoxic). For example - encoding molecules (instances) in my dataset by a vector of numbers corresponding to a) a 2D vs. b) a 3D representation of molecular structure.

  2. In both cases, I generate k-fold cross-validation results (non-overlapping folds) using Random Forest acros R repetitions of cross-validation. Furthermore, for each train/validation set pair (corresponding to a given value of R and k), I repeat model generation and validation Q times - each time, using a different seed to initialise the random number generator for Random Forest.

    E.g. for a given value of k and R (I am working in the R programming language):

    rf <- randomForest(current_train_x,current_train_y)
    predictions <- predict(rf,current_test_x)
    figure_of_merit <- fom_function(predictions,current_test_y)
  3. I can then take the arithmetic mean of all figure_of_merit values (for both sets of descriptors). I wish to get a p-value corresponding to the null-hypothesis: The mean difference in this figure of merit for both methods is zero.

Supposing I was not considering multiple randomForest RNG seeds, I could obtain such a p-value using Bouckaert and Frank's 2004 “corrected repeated k-fold cv test”.

But, how could I modify this to get a p-value for the overall mean difference???

(The only approach I can currently think of is getting one p-value per RNG seed value, then (after correcting for multiple testing using p.adjust(...)) seeing whether the p-value is statistically significant for all RNG seeds.)


I am also a chemoinformatician, so I can be wrong, but since there is no any answer, I'll put here my suggestions. Really, I don't see a problem here. The mean of random variables is also a random variable. So, you have to obtain a k x R matrix of *mean_figure_of_merit2D* (each mean is taken from Q *figure_of_merit2D*). Then transform this matrix to a vector like
(You can also go straightly to the vector form in the cross-validation cycle)
Obtain *mean_figure_of_merit3D* the same way. And then - as the Bouckaert and Frank's paper stating -
x<-mean_figure_of_merit2D - mean_figure_of_merit3D

More repetitions (R) are made - then the distribution of t will be closer to Student's t-distribution - more correct p-value will be obtained.
BUT even if you'll prove that 3D descriptors are better than 2D (or vice versa) - this inference has no value, since it concerns the prespecified dataset treated with prespecified algorithm that was run with the prespecified parameters - and thus cannot be extrapolated. I'll better recommend you to combine your descriptors together and thus increase the chance to obtain more predictive model. And one else point: since random forest algorithm is using bagging, you don't need to cross-validate your model - the out-of-bag statistics is unbiased (see http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#ooberr).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.