0
$\begingroup$

I'm confused with my data I'm currently playing with.

I have a data set which holds 58 attributes in 10000 instances. Attributes are 56 float values typically within 0 to 1. Then there is nominal attribute which tells "data class". Last one is result of data, which is nominal attribute with 3 different values.

When playing with Weka with this data set and if i do cross-validation with 1000 folds and try to classify that data with random forest, I get pretty good results, somewhat ~85% of instances are classified correctly. Same applies with Multilayer perceptron (however I haven't done it with that many folds since one fold takes ~5 minutes of time).

Problem comes when I test this data with own test set, for example last 10 instances from model, and make a new model with 9990 instances (actually I have plenty of more data, so model is still 10000 instances wide, and no test data is involved there).

Now my result is highly worse. ~40% of instances are classified now correctly. How can this be? What I don't understand here and how can I improve this results with my own test data?

$\endgroup$
4
  • 2
    $\begingroup$ Are you really using only 10 test instances? That doesn't make a lot of sense. $\endgroup$ Commented Aug 25, 2014 at 18:06
  • $\begingroup$ as far as i know, it makes that 10 instance test 1000 times. However with 100 fold test result does not change result much. Only +- 2% $\endgroup$
    – cyberseppo
    Commented Aug 25, 2014 at 18:10
  • $\begingroup$ I mean in the 3rd paragraph, not in cross-validation. You mention using 10 instances there. Is this done only once? If so, then any statistics you derive based on such a small sample can vary a lot. $\endgroup$ Commented Aug 25, 2014 at 18:14
  • $\begingroup$ No no. That's just one case. My 40% approximate is average of ~500 of those calculations. Every time fresh instances and new model. $\endgroup$
    – cyberseppo
    Commented Aug 25, 2014 at 18:18

2 Answers 2

1
$\begingroup$

It is likely that the order of your folds in your dataset has a signification.

Try to shuffle your data before splitting it between train data/valid data/test data to alleviate the bias.

$\endgroup$
1
  • $\begingroup$ Nice guess. Haven't thought about that. However in my quick 50-fold test there was the "same" 85% TP-rate. $\endgroup$
    – cyberseppo
    Commented Aug 25, 2014 at 18:25
1
$\begingroup$

So you have indication of strong overfitting to your training and cross validation set.

One guess would be that there may be a strong clustering/hierarchical structure in the data, and the independent test set is really independent, i.e. containst new clusters. E.g. you train on 10000 instances wich are 1000 repeated measurements from 10 individuals. The test set consists of unknown individuals.

$\endgroup$
2
  • $\begingroup$ Ok, two questions left; 1) Can there be strong overfit with both algorithms, multilayer perceptron and random forest? 2) How can that overfit when I test my data with 1000 split cross validation? Would it help much if I post my model .arff file? $\endgroup$
    – cyberseppo
    Commented Aug 28, 2014 at 17:16
  • $\begingroup$ Yes, with a strong clustering in the data which is not taken into account by be splitting for cross validation but by the external independent test such an overfitting is possible. Random forests can overfit in such a fashion if their internal resampling is not aware of the clustering (i.e. that would most probably be the case unless you've modified your algorithm in that respect). The custering could effectively knock out the anti-overfitting strategy of the random forest. In general it is difficult to guard against such overfitting unless you know the cause. $\endgroup$ Commented Sep 3, 2014 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.