So I am currently trying out multiple techniques of modelling a predictive model. I have 30 attributes (numeric) as input and 1 output (also numeric).

At first I used a Linear regression which did succeed and modeled the problem quite well. It scored R^2 = 0,736 in cross valiation and when used on a real non-training dataset it performed as expected predicting the results reasonably well.


Now I am trying to use a Regression Forest to model the problem and see if I can get better results. I do a 10-fold cross validation with random sampling to train the forest and get what seems to me like really good results:

Scores Scatter Plot Prediction (Y) vs Target (X)

But if I let this model run over unseen data it completely fails. Which I find kind of misleading as my cross-validation looked so promising.

Stats unseen Scatter unseen Prediction (Y) vs Target (X)


Can someone explain to me why cross validation and unseen data can yield such dramatically different results?

Further Info

I am using Knime IDE and their Tree Ensemble Learner/Predictor nodes right now. The model used for predicting the "unseen" data is the the last produced by the cross validation. (not anymore see schematic) The Tree Ensemble does not sample data for training again and trains 100 trees. To get the final prediction of the trees, a simple mean is used on all trees' outputs.

For more Information about the learner and its settings see Knime online Docs

Graphs of the predictions: predictions

As can be seen, the last fold predicts its 30% of test data very well. But when I use the same model on more test data the model completely fails.

Knime Schematic


Test and training data are extracted from the same system. The training dataset is large enough to describe the system completely

  • 1
    $\begingroup$ First answer would be overfitting. Is your training / test data completely separated? You mentioned that you used your last fold as test data?? That is very suspicious. Also are your training / test data sets very different? Do they come from two different sources? $\endgroup$ – user2974951 Oct 9 '18 at 9:12
  • $\begingroup$ I only take the model from my last fold no data. Training data and test data are completely seperate extractions from the same system (Just from diferent times) but I made sure that I have enough training data that the system can be completely described in the model. Regarding overfitting I'm gonna extract some graphs and see if I can find some indicators.but my predictions so far seem to be just a scrambled mess. $\endgroup$ – Robin B Oct 9 '18 at 9:24
  • $\begingroup$ Why do you take only the last model in the last fold? CV is done on many folds, but the results are then averaged over all of them. Also, what exactly is it that you are trying to predict here? Is thim time series data? $\endgroup$ – user2974951 Oct 9 '18 at 9:28
  • 1
    $\begingroup$ do you "preprocess" the train/test data, e.g. Z-score? $\endgroup$ – DataD'oh Oct 9 '18 at 9:59
  • 1
    $\begingroup$ Maybe you have some sort of information leakage in you CV? $\endgroup$ – sebhofer Oct 9 '18 at 11:42

you cannot use cross validation in trees. Please see "The Elements of Statistical Learning" pp. 254.

| cite | improve this answer | |
  • $\begingroup$ With other fitting methods like trees, cross-validation (...) can underestimate the true error by 10%, because the search for best tree is strongly affected by the validation set. In these situations only a separate test set will provide an unbiased estimate of test error. I assume this is the paragraph in question? Can you iterate on how this answers the question in more detail? $\endgroup$ – Robin B Oct 29 '19 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.