I have often found in practice, using nonlinear regression techniques such as feedforward neural nets or random forests, that the resulting actual-vs-fitted plot (on training set) seems obviously sub-optimal, eg. like this:

problematic actual vs. fitted plot

Here the model under-estimates high values of $y$, and under-estimates low values. It would seem natural to me "shift" these predictions using linear regression of the observed $y$ over the predicted $y$:

  • I'd first run a nonlinear regression model, and obtain a first training-set prediction $\hat y_{train} = f(X_{train})$

  • then I'd run a linear regression of $y_{train}$ on $\hat y_{train}$, and get $y_{train} = \hat a + \hat b \times \hat y_{train} + \hat \epsilon$

  • then for any prediction set $X_{test}$, I'd output a re-scaled prediction: $\hat y_{test} = \hat a + \hat b \times f(X_{test})$.

It seems obvious that this yields lower error on the training set, but does anyone know whether it hurts generalization error?

  • $\begingroup$ Have you cross-validated with and without it? What about holding a separate "calibration" set of data that learns the linear model from the $\hat y_{held}$? $\endgroup$ – Jessica Mick Aug 20 '14 at 17:25
  • $\begingroup$ Thanks, well on this particular dataset the linear post-treatment gives a higher cross-validated prediction error, but I was wondering if there was perhaps a more general answer. $\endgroup$ – jubo Aug 21 '14 at 2:27
  • $\begingroup$ I don't know much about neural nets and random forests, but I'm surprised to see a fit like that. Hopefully someone can answer with an explanation of why something like this would happen. $\endgroup$ – shadowtalker Aug 29 '14 at 11:25

A lot would depend on the data set itself. From your graph it appears that the RF(or NN) is missing on training some information. In such cases, you must almost definitely do what you are proposing. Yet another approach would be to apply the same RF (or NN) methods on the residuals and train on learning those errors to make a second order prediction. You can extend it onwards to a 3rd, 4th level where you keep learning on the residuals, until you are left with white noise. But the true test for such methods is a k-way cross validation where you keep a hold-out set of data and test on it.

  • $\begingroup$ Thanks, could you be more specific about "the RF is missing on some information"? Also, do you know of any literature on this topic of post-treatment? And as such it doesn't seem too crazy for the RF/NN to shrink extreme values towards the mean, for better generalization... $\endgroup$ – jubo Aug 26 '14 at 16:06

This might sound silly but has your algo converged yet? There is a smaller MSE out there and the optimisation failed to find it. It can be

  1. The algo hasn't converged
  2. The local minimum it found is not good enough. So you can try multi start point.
  3. How is your learning rate?
  • $\begingroup$ I agree, there is obviously a better MSE out there, which the optimization process failed to find, probably stuck in a local optimum. But I tried tweaking the hyperparameters, nothing doing, still these sub-optimal predictions... $\endgroup$ – jubo Aug 29 '14 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.