# Comparing performance of drugs with different variances and sample sizes

I am considering the following hypothetical example. Suppose that I a have a classification model that predicts whether a patient will survive or not. The features of this classification model are $$\mathbf{x} = (x_1,\dots,x_n)$$ which represent quantities related to a drug and some other variables like hospital condition, doctor care, etc.

Suppose I have 1000 drugs and I have $$n_i$$ samples of the features $$\mathbf{x}$$ for each drug, $$i=1,\dots,1000$$. Using the $$n_i$$ feature samples for each drug, I can then obtain $$n_i$$ samples of $$p_i$$, the probability that a patient will survive or not.

Using this data, I then want to rank the drugs from best to worst. Is there a "metric" that I can compute based on these samples? Naturally, I'd like to compare the mean of $$p_i,i=1,\dots,1000$$ but the standard deviations are different and the sample size $$n_i$$ are also different. The difference in the sample size can be up to 1-3 orders of magnitude, i.e. one drug can have 10 samples while another can have 2000 samples.

I was browsing online and read that a Welch's t-test is applicable to my situation. However, I have a few issues:

1. t-test is done pairwise. Given that I'm comparing 1000 drugs, it would be hard to produce a ranking or even obtain the top 10 drugs, for example.
2. The t-test only compares whether $$\mu_X = \mu_Y$$ or $$\mu_X \neq \mu_Y$$ where $$X$$ and $$Y$$ are 2 groups. How can we say that one group has a larger mean than the other?

Any suggestions/references on this matter? Thanks!

The only circumstance where it is realistic to think of comparing a thousand drugs would be a pre-clinical screening program and the appropriate advice should relate to assay-related issues and cost-benefit analyses with consideration of what happens to the drugs that are classed as 'hits'. It is not a purely statistical issue.

If you perform a t-test (Welch or standard), or any other significance test (e.g. a permutations test), you will have a list of P-values that would allow you to rank the drugs on the basis of the statistical evidence in their respective datasets against their respective null hypotheses. That might be satisfactory for a statistics student, but it would be terrible for a clinician because it would allow a drug for which there is strong evidence for a minor clinical benefit to rank higher than a drug with moderate evidence for a huge benefit.

You need criteria for ranking that include the features that are important to the types of inference that you need to make. If you choose the mean benefit then you do not need any statistical procedure beyond calculation of the means.

Ranking drugs should be more complicated than you might think. For example, some drugs benefit only a subset of patients, and some cause harm to only a subset. Treating the patients as a single population might lead to a drug that is curative to some people ranking lower than a drug that brings a minor benefit to most patients. Another example of complexity is the fact that drugs may need different doses and regimens in different people.