I want to come up with a way to get how confident I am in my predictions. I am not using a Bayesian model so I was thinking a bootstrap confidence interval would be good:
I would re-sample my original training set, train on it and predict. And repeat that n number of times. I would then gather the n predictions and I could get some bootstrap intervals on my prediction.
The way that I interpret that would be: the larger the interval, the more sensitive my prediction is to the variation in the training data, which would in turn mean that I am fairly uncertain about my prediction. The smaller the interval would mean the opposite.
The questions are:
1) Is this a sensible thing to do to start with? 2) Is there any other interpretation that could be made? In this case, it is not true that for a 95% interval say, I would expect my interval to contain the truth 95% of the time, correct?