So I am currently trying out multiple techniques of modelling a predictive model. I have 30 attributes (numeric) as input and 1 output (also numeric).
At first I used a Linear regression which did succeed and modeled the problem quite well. It scored
R^2 = 0,736 in cross valiation and when used on a real non-training dataset it performed as expected predicting the results reasonably well.
Now I am trying to use a Regression Forest to model the problem and see if I can get better results. I do a 10-fold cross validation with random sampling to train the forest and get what seems to me like really good results:
But if I let this model run over unseen data it completely fails. Which I find kind of misleading as my cross-validation looked so promising.
Can someone explain to me why cross validation and unseen data can yield such dramatically different results?
I am using Knime IDE and their Tree Ensemble Learner/Predictor nodes right now.
The model used for predicting the "unseen" data is the the last produced by the cross validation. (not anymore see schematic)
The Tree Ensemble does not sample data for training again and trains 100 trees.
To get the final prediction of the trees, a simple mean is used on all trees' outputs.
For more Information about the learner and its settings see Knime online Docs
As can be seen, the last fold predicts its 30% of test data very well. But when I use the same model on more test data the model completely fails.
Test and training data are extracted from the same system. The training dataset is large enough to describe the system completely