Suppose I am building a model for regression problems. I am quite curious about the following questions:
Are there relevant theories that can confirm/disprove the following intuition: we should penalize a point prediction that has larger prediction interval(relative to other points), assuming we can get some kind of measure for the prediction interval (whether the model is linear regression or random forest).
If so, what kind of penalization is appropriate? For example, shrinking towards sample mean? Or shrinking towards a constant like 0 under the appropriate context? Other methods?
I tried googling a bit but didn't find much. Maybe it all depends. Any relevant information is appreciated. Thanks.
EDIT: An example would be that, given an already trained random forest and a new input X1 and its prediction interval is very small, so it's a sign of high model confidence, so I am happy to use the model estimate; while for another input X2, the prediction interval is large (every tree is giving wildly different predictions), so maybe I should just shrink it towards say, sample mean, and perhaps ultimately it's a better prediction.