Ranking the Outputs of Statistical Models I have the following question about Ranking the Outputs of Statistical Models.

*

*In the case of a regression model with a continuous response variable : Suppose the model makes predictions on 3 new observations (e.g. failure time for these 3 observations : observation #1 expected to fail in 10 days plus minus 5 days, observation #2 expected to fail in 50 days plus minus 20 days, observation #3 expected to fail in  200 days plus minus 100 days) - based on the confidence intervals of the predicted values of these 3 observations, are we allowed to rank these times? For example, could we say that observation #3 is likely to fail after observation #1?



*

*In the case of a classification model with a categorical response variable: Suppose the model predicts that observation #1 belongs to "class 1" with a probability of 0.9 and that observation #2 belongs to "class 0" with a probability of 0.53 - based on the confidence intervals, are we allowed to rank these probabilities? For example, could we say that the prediction made for observation #1 is more "certain" than the prediction made for observation #2?

My Question: Can the predictions made by statistical models be ranked as the way I described?
Thanks!
 A: 
based on the confidence intervals of the predicted values of these 3 observations, are we allowed to rank these times? For example, could we say that observation #3 is likely to fail after observation #1?

The marginal distributions can give you some idea about which variable is larger but you do not know everything. Below you see an example with two different joint distributions for a pair of variables x,y. The marginal distributions are the same but the number of times that you have $x>y$ is not the same.
In the example you get that more often $x>y$ than $x<y$. But in some situations it might be possible that the mean or median of one distribution $x$ is larger than the the mean or median of the distribution of $y$ but that you get more often $y>x$.


For example, could we say that the prediction made for observation #1 is more "certain" than the prediction made for observation #2?

You could indeed say that the predictions are more "certain" if the values are closer to zero or one. A related concept is information entropy which would be computed as
$$H = \sum_{\text{for all classes $i$}} p_i \log p_i = p \log(p) + (1-p) \log(1-p)$$
This means that the model makes a more 'certain' prediction. But, the model could be bad and make a lot of false positive or false negative errors. We may not be certain about the model.
A: Your question has two separate aspects - one is the interpretation of the outputs of statistical models and the other one is the ranking of random variables based on their probability distribution.
It is easiest to answer the kind of questions you are describing in a Bayesian framework, since in that case you can describe everything probabilistically - that is, you actually get a probability distribution for every prediction (as depicted in your plot). so for example if $t_1$ and $t_2$ are the failure times of two observations you can use those distributions to calculate for example $P(t_1 > t_2) = \int_{t_1>t_2}f_1(t_1)f_2(t_2)dt_1dt_2$ which will tell you how likely it is for $t_1$ to happen after $t_2$.
However just as with any random variables there is no unique way of ranking the outputs. You could also calculate for example the expected values $\mathbb E[t_1]$ and $\mathbb E[t_2]$, and it is possible to have at the same time $\mathbb E[t_1] > \mathbb E[t_2]$ and $P(t_2 > t_1) > 1/2$, that is, the expected value of $t_1$ can be larger but $t_2 > t_1$ is more likely than $t_1 > t_2$. So, you will have to decide based on which criterion you want to rank them (this is somewhat similar to the analysis of risk vs. expected value in
finance, see e.g. Mean-variance analysis).
In a classification case where the model outputs probabilities this is more straightforward, and it does makes sense to rank based on the level of certainty. However keep in mind that usually those models don't perform a full Bayesian calculation so there can be issues with the Calibration of those probabilities.
