In classification problems, "non-probabilistic" machine learning models such as boosted trees, neural networks, etc. are known to produce poorly-calibrated class scores, which aren't suitable for use as posterior probability estimates.
However, are the rankings of scores still generally valid?
For example, consider a classification neural network with 5 outputs. Is it valid to treat the output scores as ranks? Suppose the model predicts scores of A:0.64, B:0.23, C:0.10, D:0.02, and E:0.01. Is it valid to say that classes A-C are the "top 3" model predictions? Is it valid to say that class B is more probable than class C for this prediction?
I have done this many times in my own work, and I haven't seen it produce bad results. But I have also never considered if there are known problems with this procedure, either theoretical or empirical.