# Why is my cross validation failing?

## Intro

So I am currently trying out multiple techniques of modelling a predictive model. I have 30 attributes (numeric) as input and 1 output (also numeric).

At first I used a Linear regression which did succeed and modeled the problem quite well. It scored R^2 = 0,736 in cross valiation and when used on a real non-training dataset it performed as expected predicting the results reasonably well.

## Problem

Now I am trying to use a Regression Forest to model the problem and see if I can get better results. I do a 10-fold cross validation with random sampling to train the forest and get what seems to me like really good results:

Prediction (Y) vs Target (X)

But if I let this model run over unseen data it completely fails. Which I find kind of misleading as my cross-validation looked so promising.

Prediction (Y) vs Target (X)

## Question

Can someone explain to me why cross validation and unseen data can yield such dramatically different results?

## Further Info

I am using Knime IDE and their Tree Ensemble Learner/Predictor nodes right now. The model used for predicting the "unseen" data is the the last produced by the cross validation. (not anymore see schematic) The Tree Ensemble does not sample data for training again and trains 100 trees. To get the final prediction of the trees, a simple mean is used on all trees' outputs.

For more Information about the learner and its settings see Knime online Docs

Graphs of the predictions:

As can be seen, the last fold predicts its 30% of test data very well. But when I use the same model on more test data the model completely fails.

## Knime Schematic

Test and training data are extracted from the same system. The training dataset is large enough to describe the system completely

• First answer would be overfitting. Is your training / test data completely separated? You mentioned that you used your last fold as test data?? That is very suspicious. Also are your training / test data sets very different? Do they come from two different sources? Commented Oct 9, 2018 at 9:12
• I only take the model from my last fold no data. Training data and test data are completely seperate extractions from the same system (Just from diferent times) but I made sure that I have enough training data that the system can be completely described in the model. Regarding overfitting I'm gonna extract some graphs and see if I can find some indicators.but my predictions so far seem to be just a scrambled mess. Commented Oct 9, 2018 at 9:24
• Why do you take only the last model in the last fold? CV is done on many folds, but the results are then averaged over all of them. Also, what exactly is it that you are trying to predict here? Is thim time series data? Commented Oct 9, 2018 at 9:28
• do you "preprocess" the train/test data, e.g. Z-score?
– Krrr
Commented Oct 9, 2018 at 9:59
• Maybe you have some sort of information leakage in you CV? Commented Oct 9, 2018 at 11:42

• With other fitting methods like trees, cross-validation (...) can underestimate the true error by 10%, because the search for best tree is strongly affected by the validation set. In these situations only a separate test set will provide an unbiased estimate of test error.  I assume this is the paragraph in question? Can you iterate on how this answers the question in more detail? Commented Oct 29, 2019 at 9:38