# Linear post-treatment of nonlinear regression

I have often found in practice, using nonlinear regression techniques such as feedforward neural nets or random forests, that the resulting actual-vs-fitted plot (on training set) seems obviously sub-optimal, eg. like this:

Here the model under-estimates high values of $y$, and under-estimates low values. It would seem natural to me "shift" these predictions using linear regression of the observed $y$ over the predicted $y$:

• I'd first run a nonlinear regression model, and obtain a first training-set prediction $\hat y_{train} = f(X_{train})$

• then I'd run a linear regression of $y_{train}$ on $\hat y_{train}$, and get $y_{train} = \hat a + \hat b \times \hat y_{train} + \hat \epsilon$

• then for any prediction set $X_{test}$, I'd output a re-scaled prediction: $\hat y_{test} = \hat a + \hat b \times f(X_{test})$.

It seems obvious that this yields lower error on the training set, but does anyone know whether it hurts generalization error?

• Have you cross-validated with and without it? What about holding a separate "calibration" set of data that learns the linear model from the $\hat y_{held}$? – Jessica Mick Aug 20 '14 at 17:25
• Thanks, well on this particular dataset the linear post-treatment gives a higher cross-validated prediction error, but I was wondering if there was perhaps a more general answer. – jubo Aug 21 '14 at 2:27
• I don't know much about neural nets and random forests, but I'm surprised to see a fit like that. Hopefully someone can answer with an explanation of why something like this would happen. – shadowtalker Aug 29 '14 at 11:25