I am considering the following hypothetical example. Suppose that I a have a classification model that predicts whether a patient will survive or not. The features of this classification model are $\mathbf{x} = (x_1,\dots,x_n)$ which represent quantities related to a drug and some other variables like hospital condition, doctor care, etc.
Suppose I have 1000 drugs and I have $n_i$ samples of the features $\mathbf{x}$ for each drug, $i=1,\dots,1000$. Using the $n_i$ feature samples for each drug, I can then obtain $n_i$ samples of $p_i$, the probability that a patient will survive or not.
Using this data, I then want to rank the drugs from best to worst. Is there a "metric" that I can compute based on these samples? Naturally, I'd like to compare the mean of $p_i,i=1,\dots,1000$ but the standard deviations are different and the sample size $n_i$ are also different. The difference in the sample size can be up to 1-3 orders of magnitude, i.e. one drug can have 10 samples while another can have 2000 samples.
I was browsing online and read that a Welch's t-test is applicable to my situation. However, I have a few issues:
- t-test is done pairwise. Given that I'm comparing 1000 drugs, it would be hard to produce a ranking or even obtain the top 10 drugs, for example.
- The t-test only compares whether $\mu_X = \mu_Y$ or $\mu_X \neq \mu_Y$ where $X$ and $Y$ are 2 groups. How can we say that one group has a larger mean than the other?
Any suggestions/references on this matter? Thanks!