I'm confused with my data I'm currently playing with.
I have a data set which holds 58 attributes in 10000 instances. Attributes are 56 float values typically within 0 to 1. Then there is nominal attribute which tells "data class". Last one is result of data, which is nominal attribute with 3 different values.
When playing with Weka with this data set and if i do cross-validation with 1000 folds and try to classify that data with random forest, I get pretty good results, somewhat ~85% of instances are classified correctly. Same applies with Multilayer perceptron (however I haven't done it with that many folds since one fold takes ~5 minutes of time).
Problem comes when I test this data with own test set, for example last 10 instances from model, and make a new model with 9990 instances (actually I have plenty of more data, so model is still 10000 instances wide, and no test data is involved there).
Now my result is highly worse. ~40% of instances are classified now correctly. How can this be? What I don't understand here and how can I improve this results with my own test data?