2
$\begingroup$

Suppose I am building a model for regression problems. I am quite curious about the following questions:

  1. Are there relevant theories that can confirm/disprove the following intuition: we should penalize a point prediction that has larger prediction interval(relative to other points), assuming we can get some kind of measure for the prediction interval (whether the model is linear regression or random forest).

  2. If so, what kind of penalization is appropriate? For example, shrinking towards sample mean? Or shrinking towards a constant like 0 under the appropriate context? Other methods?

I tried googling a bit but didn't find much. Maybe it all depends. Any relevant information is appreciated. Thanks.

EDIT: An example would be that, given an already trained random forest and a new input X1 and its prediction interval is very small, so it's a sign of high model confidence, so I am happy to use the model estimate; while for another input X2, the prediction interval is large (every tree is giving wildly different predictions), so maybe I should just shrink it towards say, sample mean, and perhaps ultimately it's a better prediction.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

If you assume that those predictors with limited predictive power in the model will also be those with large confidence intervals (which follows logically), some form of penalised regression is what you are looking for.

In penalised regression, coefficients that add little predictive value are shrunk to zero. The most common penalised regressions are lasso and ridge regression, on which there is a lot of literature available..

Hope this helps.

Improve this answer
$\endgroup$
10
  • $\begingroup$ I think the goal is to penalize predictions (greater loss function) where the model is saying something like, "I'm predicting 5, but I really have no idea," rather than anything to do with standard errors of regression parameters. $\endgroup$
    – Dave
    Commented Jul 27, 2019 at 21:32
  • $\begingroup$ Ah right, I see. Well in that case I must say I don't really understand how you would want to penalise - the model gives the most logical estimate given the data.. If it really has no idea based on regressors, it should give the most logical 'container' value (e.g. the mean).. $\endgroup$
    – Mark Verhagen
    Commented Jul 27, 2019 at 21:38
  • $\begingroup$ For my example above, let's say that the mean is 0. Then the error would be 5, and the squared error would be 25. However, if the model really has no idea if it should predict 5 instead of 20, perhaps the contribution to the loss function should be greater than 25. (I don't know how to do this, but I think iwbabn is after something like this.) $\endgroup$
    – Dave
    Commented Jul 27, 2019 at 21:53
  • $\begingroup$ I think this is a bit problematic though, because for example the estimate of the mean / intercept in a standard model fundamentally depends on the loss being exactly that: the square of the error. I wonder how toying with the loss function based on confidence intervals would affect the integrity of the estimates.. Also difficult I suppose to make a confiidence (or unconfidence in this case) indicationi which requires coefficient estimates which then depend on an initial fixed loss function. I suppose it should be some iterative process then..? $\endgroup$
    – Mark Verhagen
    Commented Jul 27, 2019 at 22:01
  • $\begingroup$ It's routine to use loss functions other than MSE. That's not an issue. You're right that we're no longer doing OLS, but that's okay. There are lots of other ways to do regression. $\endgroup$
    – Dave
    Commented Jul 27, 2019 at 22:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.