4
$\begingroup$

I have often found in practice, using nonlinear regression techniques such as feedforward neural nets or random forests, that the resulting actual-vs-fitted plot (on training set) seems obviously sub-optimal, eg. like this:

problematic actual vs. fitted plot

Here the model under-estimates high values of $y$, and under-estimates low values. It would seem natural to me "shift" these predictions using linear regression of the observed $y$ over the predicted $y$:

  • I'd first run a nonlinear regression model, and obtain a first training-set prediction $\hat y_{train} = f(X_{train})$

  • then I'd run a linear regression of $y_{train}$ on $\hat y_{train}$, and get $y_{train} = \hat a + \hat b \times \hat y_{train} + \hat \epsilon$

  • then for any prediction set $X_{test}$, I'd output a re-scaled prediction: $\hat y_{test} = \hat a + \hat b \times f(X_{test})$.

It seems obvious that this yields lower error on the training set, but does anyone know whether it hurts generalization error?

Improve this question
$\endgroup$
3
  • $\begingroup$ Have you cross-validated with and without it? What about holding a separate "calibration" set of data that learns the linear model from the $\hat y_{held}$? $\endgroup$
    – Jessica Collins
    Commented Aug 20, 2014 at 17:25
  • $\begingroup$ Thanks, well on this particular dataset the linear post-treatment gives a higher cross-validated prediction error, but I was wondering if there was perhaps a more general answer. $\endgroup$
    – jubo
    Commented Aug 21, 2014 at 2:27
  • $\begingroup$ I don't know much about neural nets and random forests, but I'm surprised to see a fit like that. Hopefully someone can answer with an explanation of why something like this would happen. $\endgroup$
    – shadowtalker
    Commented Aug 29, 2014 at 11:25

2 Answers 2

Reset to default
1
$\begingroup$

A lot would depend on the data set itself. From your graph it appears that the RF(or NN) is missing on training some information. In such cases, you must almost definitely do what you are proposing. Yet another approach would be to apply the same RF (or NN) methods on the residuals and train on learning those errors to make a second order prediction. You can extend it onwards to a 3rd, 4th level where you keep learning on the residuals, until you are left with white noise. But the true test for such methods is a k-way cross validation where you keep a hold-out set of data and test on it.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, could you be more specific about "the RF is missing on some information"? Also, do you know of any literature on this topic of post-treatment? And as such it doesn't seem too crazy for the RF/NN to shrink extreme values towards the mean, for better generalization... $\endgroup$
    – jubo
    Commented Aug 26, 2014 at 16:06
1
$\begingroup$

This might sound silly but has your algo converged yet? There is a smaller MSE out there and the optimisation failed to find it. It can be

  1. The algo hasn't converged
  2. The local minimum it found is not good enough. So you can try multi start point.
  3. How is your learning rate?
Improve this answer
$\endgroup$
1
  • $\begingroup$ I agree, there is obviously a better MSE out there, which the optimization process failed to find, probably stuck in a local optimum. But I tried tweaking the hyperparameters, nothing doing, still these sub-optimal predictions... $\endgroup$
    – jubo
    Commented Aug 29, 2014 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.