I am modelling time to failure of some units. All units are made my the same manufacturer and there are no recorded covariates to distinguish between the units.
This means that the central, $95\%$ say, remaining useful life (RUL) prediction intervals for unit $i$ and unit $j$ are the same if unit $i$ and unit $j$ are the same age.
I would like to assess the predictive capability of my model. I initially thought I could investigate the model's ability to predict remaining useful life.
Suppose all units are new (age 0) at the time of the first prediction and suppose I have 1000 units and suppose initially that I have observed all failure times (no right censoring).
If my model is well calibrated I should expect the coverage probability of the central (or any other) $95\%$ prediction interval to be approximately 0.95. I.e., I should expect the true observations to lie in the central $95\%$ prediction interval approximately $95\%$ of the time.
More generally, I should expect the probability integral transform (PIT) to look uniform. This would mean that approximately $p\%$ of my observations fall below the $p$th quantile of my model.
This is a common way to assess model fit.
Now suppose I have 1000 units (all age 0) but I only observe 50 failures (perhaps I end the test early) and hence 950 units are right censored.
I can only assess whether the true value lies within the prediction interval for the 50 units that fail, and hence I have discarded the 950 right censored units.
I find that my PIT is not uniform and consequently the coverage probability of the central $95\%$ prediction interval is not approximately 0.95.
Initially, I worried that my model fit poorly to the data. However, I think this is to be expected.
All prediction intervals are the same (since all units are new and identical). Suppose the $95\%$ central prediction interval is $[L,U]$. If I had observed all failures, I should expect approximately 25 units ($2.5\%$ of units) to fall below $L$ [i.e. below the lower $2.5\%$ quantile].
I have only observed the earliest 50 failures. So, when I discard the 950 units with no failure times, I find that the coverage probability is very low, which I now think is to be expected.
More specifically, I think I should expect the coverage probability to be 0.5. I.e., I should expect 25 units to fall below $L$ (the earliest 25 failures) and I should expect 25 failures to lie within the prediction interval. Then, since I discarded the 950 right censored units, the coverage probability is expected to be 25/50 = 0.5.
Does this make sense? If yes, are there adjustments I can make to incorporate the unobserved right censored failure times?