I'm really new to ML so apologies if this is obvious to those with an ML. background. I have two experiments where we analysed intestinal bacteria composition in genetically modified animals with controls in one and another where we associated germ-free (animals bred completely sterile) with the bacteria from the first experiment, also with controls (total n=198).

Basically, nearly all of the genetically modified animals will get sick, however in the associated group only ~60% get sick.

Although these are two separate experiments, the data was merged and reanalysed, as due to the nature of these analyses it is not valid to compare two datasets analysed separately.

I wish to use a subset of data (n=42) from the first experiment to predict which of the germ-free associated group will get sick. My current approach is training a random forest on the aforementioned subset with leave-one-out cross validation, and then using a subset of data from the second experiment (n=32) as the test set. Currently my model seems to predict outcome with an accuracy of 84%.

My question is, is this a valid approach? As far as I know you should not choose which data goes into the test set when you are splitting a single dataset into test and training, however in this instance they are technically two datasets.


1 Answer 1


My understanding of the problem is as follows: There are two datasets 1. Genetically modified, with 100% getting sick 2. associated group, with 60% getting sick. What is the goal of the machine learning algorithm? What is the test input that will be providing at a later stage to the algorithm?

  1. If the test input is going to provide only the genetically modified species as input, then it makes sense to use only those samples for both training and testing (cross validation).
  2. If the test input is going to use associated group species as input, then it makes sense to use only those samples related to the associated group for both training and testing (cross validation).
  3. If the test input is using any of the two types, then the test training set has to be mix of samples from both sets for more accurate predictions. Even though, biologically, the experiments might be different, the features extracted may be same for the machine learning. And the algorithm should be able to identify the combination of features that can do appropriate predictions.

In this specific case, for the test set used (n=32), you might get high accuracy. Since the samples from this experiment are not part of the training set, the predictions will be more biased to the first experimental data. Another sample test set extracted from the associated group samples might yield much lower accuracy. The training set ideally should take representative samples from both the experiments. And typically, we train the models with 70% - 75% of available data randomly sampled. The rest 30% - 25% will be used as test data for validation.

  • $\begingroup$ Thank-you. The associated group are also genetically modified in the same manner but they are associated with the bacteria from the original experiment, so theoretically they should be the same or only slightly different. The goal is to determine if we can predict the phenotype, tumor or non-tumor, based on their bacterial composition. Currently, there is no plan for further input. As I understand an ideal test set would be independent never seen before data. In this case the data should be the same or at the very least highly similar, thus is this then not an 'ideal' testing set? $\endgroup$
    – adamsorbie
    Dec 6, 2017 at 16:33
  • $\begingroup$ The ideal test set would be a combination of data from both the experiments. $\endgroup$
    – Siva Kumar
    Dec 7, 2017 at 4:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.