I'm confused with my data I'm currently playing with.

I have a data set which holds 58 attributes in 10000 instances. Attributes are 56 float values typically within 0 to 1. Then there is nominal attribute which tells "data class". Last one is result of data, which is nominal attribute with 3 different values.

When playing with Weka with this data set and if i do cross-validation with 1000 folds and try to classify that data with random forest, I get pretty good results, somewhat ~85% of instances are classified correctly. Same applies with Multilayer perceptron (however I haven't done it with that many folds since one fold takes ~5 minutes of time).

Problem comes when I test this data with own test set, for example last 10 instances from model, and make a new model with 9990 instances (actually I have plenty of more data, so model is still 10000 instances wide, and no test data is involved there).

Now my result is highly worse. ~40% of instances are classified now correctly. How can this be? What I don't understand here and how can I improve this results with my own test data?

  • 2
    $\begingroup$ Are you really using only 10 test instances? That doesn't make a lot of sense. $\endgroup$ – Marc Claesen Aug 25 '14 at 18:06
  • $\begingroup$ as far as i know, it makes that 10 instance test 1000 times. However with 100 fold test result does not change result much. Only +- 2% $\endgroup$ – cyberseppo Aug 25 '14 at 18:10
  • $\begingroup$ I mean in the 3rd paragraph, not in cross-validation. You mention using 10 instances there. Is this done only once? If so, then any statistics you derive based on such a small sample can vary a lot. $\endgroup$ – Marc Claesen Aug 25 '14 at 18:14
  • $\begingroup$ No no. That's just one case. My 40% approximate is average of ~500 of those calculations. Every time fresh instances and new model. $\endgroup$ – cyberseppo Aug 25 '14 at 18:18

It is likely that the order of your folds in your dataset has a signification.

Try to shuffle your data before splitting it between train data/valid data/test data to alleviate the bias.

  • $\begingroup$ Nice guess. Haven't thought about that. However in my quick 50-fold test there was the "same" 85% TP-rate. $\endgroup$ – cyberseppo Aug 25 '14 at 18:25

So you have indication of strong overfitting to your training and cross validation set.

One guess would be that there may be a strong clustering/hierarchical structure in the data, and the independent test set is really independent, i.e. containst new clusters. E.g. you train on 10000 instances wich are 1000 repeated measurements from 10 individuals. The test set consists of unknown individuals.

  • $\begingroup$ Ok, two questions left; 1) Can there be strong overfit with both algorithms, multilayer perceptron and random forest? 2) How can that overfit when I test my data with 1000 split cross validation? Would it help much if I post my model .arff file? $\endgroup$ – cyberseppo Aug 28 '14 at 17:16
  • $\begingroup$ Yes, with a strong clustering in the data which is not taken into account by be splitting for cross validation but by the external independent test such an overfitting is possible. Random forests can overfit in such a fashion if their internal resampling is not aware of the clustering (i.e. that would most probably be the case unless you've modified your algorithm in that respect). The custering could effectively knock out the anti-overfitting strategy of the random forest. In general it is difficult to guard against such overfitting unless you know the cause. $\endgroup$ – cbeleites unhappy with SX Sep 3 '14 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.