1
$\begingroup$

For my research purposes, I am trying to eliminate the randomness in k-fold cross validation. My goal is to conduct cross validation where the first 10% from the dataset is the first fold so that the order of the instances prediction results is printed as their order in the dataset. And the final result after the 10 folds cross validation is the average of the 10 rounds. I tried to modify the following code posted by weka as follows:

package pkg10foldcrossvalidation;

import java.io.BufferedReader;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.Arrays;
import weka.core.Instances;
import weka.classifiers.bayes.NaiveBayes;
import weka.core.Utils;
import weka.classifiers.Classifier;
import weka.classifiers.Evaluation;
import weka.classifiers.evaluation.Prediction;

public class CrossValidationSingleRunNB {


  public static void main(String[] args) throws Exception {

// loads data and set class index
    BufferedReader br = null;
    br = new BufferedReader(new FileReader("C:\\Prediction Results on the testing set\\Dataset.arff"));

    Instances data = new Instances(br);
    data.setClassIndex(data.numAttributes() - 1);
    br.close();


    // classifier
    NaiveBayes cls = new NaiveBayes();
    String[] options = {"-D"};  
    cls.setOptions(options);
    cls.buildClassifier(data);
    System.out.println(cls.getCapabilities());   
    System.out.println(cls.globalInfo());
    System.out.println(Arrays.toString(cls.getOptions()));
    System.out.println(cls.listOptions());
    System.out.println(cls.useKernelEstimatorTipText()+"\n");
    System.out.println(cls.getUseKernelEstimator());


    // other options
    int seed  = 0;
    int folds = 10;
    Instances Data = new Instances(data);


    // perform cross-validation
    Evaluation eval = new Evaluation(Data);
    ArrayList<Prediction> predictions;
    int counter =1;
        String plus ="+";
        String minus ="-";

    for (int n = 0; n < folds; n++) {
      Instances train = Data.trainCV(folds, n);
      Instances test = Data.testCV(folds, n);

      // build and evaluate classifier
      Classifier clsCopy = NaiveBayes.makeCopy(cls);
      clsCopy.buildClassifier(train);
      eval.evaluateModel(clsCopy, test);
      predictions = eval.predictions();

        for (int i = 0, trainDataSize = test.size(); i < trainDataSize; i++) {

            Prediction prediction = predictions.get(i);

              if(prediction.actual()==prediction.predicted()){
              //System.out.println("Instance "+counter+": " +"Actual: "+prediction.actual()+"  Prediction: "+prediction.predicted());;
              System.out.println("Instance "+counter+": " +minus);;
              counter++;
              }
              else{
              System.out.println("Instance "+counter+": " +plus);;
              counter++;
              }

        }

      System.out.println("--------------------------------");
    // Model summary for each round
      System.out.println(eval.toSummaryString("\nResults\n======\n", true));
        System.out.println(eval.toClassDetailsString());
        System.out.println("Results For Class -1- ");
        System.out.println("Precision=  " + eval.precision(0));
        System.out.println("Recall=  " + eval.recall(0));
        System.out.println("F-measure=  " + eval.fMeasure(0));
        System.out.println("Results For Class -2- ");
        System.out.println("Precision=  " + eval.precision(1));
        System.out.println("Recall=  " + eval.recall(1));
        System.out.println("F-measure=  " + eval.fMeasure(1));
        System.out.println(eval.toMatrixString());
        System.out.println("\n \n");

    }

    // output evaluation
    System.out.println("\n___________________________________________________________________\n");
    System.out.println("=== Setup ===");
    System.out.println("Classifier: " + cls.getClass().getName() + " " + Utils.joinOptions(cls.getOptions()));
    System.out.println("Dataset: " + data.relationName());
    System.out.println("Folds: " + folds);
    System.out.println("Seed: " + seed);
    System.out.println();
    System.out.println(eval.toSummaryString("=== " + folds + "-fold Cross-validation ===", false));
    System.out.println(eval.toMatrixString());
 }
}

But I am not sure about the correctness of this approach, does it really reflect the goal I'm trying to do?

Thank you

$\endgroup$
  • 1
    $\begingroup$ I think it's the goal that is an issue - why would you want to eliminate the randomness? $\endgroup$ – mkt - Reinstate Monica Mar 30 '18 at 7:09
  • $\begingroup$ @mkt I'm comparing my approach to a previous one where they used K-fold CV in their experiment, but the order of instances should be preserved since there is a certain relationship between them, so when I print the results of prediction after K-fold CV I want it to be as its order in the original dataset. $\endgroup$ – User505 Mar 30 '18 at 7:25
  • $\begingroup$ There are two issues right now: 1.) The "K-fold cross validation without randomness" part that you're trying to describe and 2.) its Weka implementation. Right now I think that since part 1 is still confusing--it's not clear what you're trying to do and why--you're not getting any help in terms of part 2... $\endgroup$ – Steve S Mar 30 '18 at 15:00
  • $\begingroup$ @SteveS I mean in the dataset I have which consists of 6300 instances there is a logical relation between each consecutive nine instances in my research, e.g. the instances from 1 to 9 , 10 to 18,…6291 to 6300. When I print the evaluation results of the instances I want to be sure that the order is not random to know for each nine instances how many instances has correctly or incorrectly classified. $\endgroup$ – User505 Mar 30 '18 at 15:45
  • 1
    $\begingroup$ In that case it sounds like what you need to do is carry on with regular cross-validation (with randomness) except act as if you have 700 observations instead of 6300 and when you select some number 'i' for a given fold, you assign all 9 observations of the i-th group to said fold. Does that make sense? $\endgroup$ – Steve S Mar 30 '18 at 18:26
1
$\begingroup$

It seems like the solution to this problem is stratified $k$-fold CV, which is a standard approach. Stratified $k$-fold CV adapts ordinary, random CV to allow objects to be grouped by strata. If some observations are all in the same strata, stratified $k$-fold CV will all put them in the same fold. If you have more strata than you have folds, then different strata will be grouped together in folds.

I have no idea if your code correctly implements this concept. If you want someone to check your code, you'll have to post it on a forum which provides that service.

$\endgroup$
1
$\begingroup$

Fundamentally, in order for CV not to be biased, you need to shuffle (randomly permute) all the objects in the data set before assigning them to the 10 folds. So if there are $n=100$ objects, you would assign the following first 10 objects in the list of shuffled objects to fold 1:

34,7,11,89,59,51,1,73,25,80

After training and testing with the 10 folds, shuffle the objects again, assign into 10 folds, and then repeat training and testing. Repeat ten times in order to perform "ten 10-fold CV."

You can do bookkeeping to track the true and predicted class (test results) of each object wherever it appears in testing folds. However, overall accuracy is the ratio of the sum of the diagonal counts in the confusion matrix to the total number of confusion matrix elements.

To list results for each objects using the above, you could simply use vectors like ShuffleID(100), Correct(100), Incorrect(100). The any time during training just pad with a 1 for whatever outcome is true when predicting class for each $i$th object in a test fold. Assuming ShuffleID(1)=34, simply loop through the 10 objects in each test fold using, for example:

For testfold = 1 to 10
    For i = 10*(testfold-1)+1  to 10*testfold
        If trueclass(ShuffleID(i)) = predictedclass(ShuffleID(i)) Then  //correctly classified
           Correct(ShuffleID(i)) +=1
        End If

        If trueclass(ShuffleID(i)) != predictedclass(ShuffleID(i)) Then //incorrectly classified
           Incorrect(ShuffleID(i)) +=1
        End If
    Next i
Next testfold 

Once training is complete, print out the results for each object:

For i = 1 to 100
   Print("i", Correct(i) / ( Correct(i) + Incorrect(i) ) )
Next i  
$\endgroup$
  • $\begingroup$ But the mean concern is how can I ensure the order of the instances. When I print the result I want to be sure that their order is the same as it appears in the dataset? $\endgroup$ – User505 Mar 30 '18 at 16:19
  • $\begingroup$ The OP was modified with what to do, as an example $\endgroup$ – JoleT Mar 30 '18 at 16:42
  • $\begingroup$ Thank you very much @wrtsvkrfm. I put a comment that describes the exact situation I'm facing. I put the code I have in java but I need someone expert to tell if this is really reflect my aim since the result I'm working on is extremely sensitive. Really thank you I appreciate it $\endgroup$ – User505 Mar 30 '18 at 17:07
  • $\begingroup$ " I mean in the dataset I have which consists of 6300 instances there is a logical relation between each consecutive nine instances in my research, e.g. the instances from 1 to 9 , 10 to 18,…6291 to 6300. When I print the evaluation results of the instances I want to be sure that the order is not random to know for each nine instances how many instances has correctly or incorrectly classified ". This is my comment for SteveS $\endgroup$ – User505 Mar 30 '18 at 19:19
  • $\begingroup$ I understand. You're using Weka and can't control the internals. Nevertheless, you do want to use something like the last print-out loop I showed, and my 100 would be changed to 6300. (the objects are in their original order for this last loop shown above). So maybe you're quesiton is about Weka programming help, and therefore, maybe use the Weka forum? $\endgroup$ – JoleT Mar 30 '18 at 20:05

Not the answer you're looking for? Browse other questions tagged or ask your own question.