4
$\begingroup$

This is a follow-up of the question I posted earlier.

I am assessing the two RF models which are generated using two different set of features

NF - Test_Accuracy > Training accuracy (500 features)

HF - Test_Accuracy < Training accuracy (125 features)

Testing and Training is done using independent data sets and the accuracy is derived from the average of a 5-cross validation. The difference between the models are in the number of features. I am afraid there could be a possible overfitting in one of the model (It is not clear to me which model could be overfitting, because I have used independent dataset and k-cross validation on the datasets). I would like to know what are the standard methods (tools/libraries) which can be used to assess overfitting.

$\endgroup$
9
$\begingroup$

This result does not mean that you have overfitting.

First of all, CV is more reliable than test set -- you can have (bad) luck in selecting test, what results in (pessimistic) optimistic bias with respect to true accuracy. CV effectively smooths this problem by repeating the procedure of selecting test. What's worse for using the test, RF is a stochastic algorithm and so two runs with different seed will give you different test accuracies, and the difference may be even bigger than that between CV and test.

Second, you may use standard deviation of accuracy from all CV runs to test whether:

  • Your CV accuracy is really different from this on test.
  • One of the feature sets you used is really better than the other.
$\endgroup$
  • $\begingroup$ Thanks for the insightful answer. I will try with SD of accuracy for further analysis. $\endgroup$ – Khader Shameer Dec 3 '10 at 16:16
1
$\begingroup$

A common and well-established approach, complementing @mbq's answer, is using nested cross validation [1]. Basically, it's able to infer generalized performance of your method as there's two splits in the data: training (where you train), test (where you test and gather relevant performance metrics to perform tuning) and validation (where you test your tuning on yet unseen labelled data). Set the number of folds and repeats of the inner and outer cross validation loops accordingly to the number of samples available.

To access overfit you can compare the performance in the inner loop to the outer loop. An overfitted model will perform much worse in the unseen data (i.e. validation data in our case).

Krstajic et al (2014) explain the method in details, and it's implemented in many statistical softwares and environments.

A similar approach is the so called double bootstrapping, where you perform bootstrapping in both loops instead of CV. Basically you can use any resampling strategy in either loop, given enough reasoning.

[1] Krstajic, D., Buturovic, L. J., Leahy, D. E., & Thomas, S. (2014). Cross-validation pitfalls when selecting and assessing regression and classification models. Journal of cheminformatics, 6(1), 1-15.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.