How to calculate uncertainty for predictions coming from cascade of models?

Question

I have developed a bunch of models to predict house prices. It is a 3 fold process:

1. I fit a gbm (first_model) and get the first prediction (first_pred),

2. there are some sub-models (simple lineer regression) for garage (sub_pred1), terrace(sub_pred2), energy(sub_pred3) and state of building(sub_pred4) to predict their impact. And get second_pred = first_pred * (1 + sub_pred1 + sub_pred2 + sub_pred3 + sub_pred4)

3. One last factor about quality is added to the second_pred. So main_pred = second_pred * (1 + quality) where quality in [0.1, -0.1]

Now I am interested in finding prediction intervals for the main_preds

I have been playing around conforming prediction through nonconformist library for python and I could directly use it if I had only 1 gbm model. But now I wonder what would be the best and doable way to come up with prediction intervals when having those different steps affecting the final prediction?

Thanks in advance for all helps and guidance!

Answer 1

For a complicated set of models like you have described that predict a continuous value, the most straightforward method for creating confidence intervals or prediction intervals is to use bootstrapping. There are many resources if you google "bootstrap prediction interval" including this one in R and this one in python.

I reviewed the conforming prediction package you mentioned, but I don't recommend going that route for continuous value predictions like housing prices.