I am considering the following hypothetical example. Suppose that I a have a classification model that predicts whether a patient will survive or not. The features of this classification model are $\mathbf{x} = (x_1,\dots,x_n)$ which represent quantities related to a drug and some other variables like hospital condition, doctor care, etc.

Suppose I have 1000 drugs and I have $n_i$ samples of the features $\mathbf{x}$ for each drug, $i=1,\dots,1000$. Using the $n_i$ feature samples for each drug, I can then obtain $n_i$ samples of $p_i$, the probability that a patient will survive or not.

Using this data, I then want to rank the drugs from best to worst. Is there a "metric" that I can compute based on these samples? Naturally, I'd like to compare the mean of $p_i,i=1,\dots,1000$ but the standard deviations are different and the sample size $n_i$ are also different. The difference in the sample size can be up to 1-3 orders of magnitude, i.e. one drug can have 10 samples while another can have 2000 samples.

I was browsing online and read that a Welch's t-test is applicable to my situation. However, I have a few issues:

  1. t-test is done pairwise. Given that I'm comparing 1000 drugs, it would be hard to produce a ranking or even obtain the top 10 drugs, for example.
  2. The t-test only compares whether $\mu_X = \mu_Y$ or $\mu_X \neq \mu_Y$ where $X$ and $Y$ are 2 groups. How can we say that one group has a larger mean than the other?

Any suggestions/references on this matter? Thanks!


1 Answer 1


The only circumstance where it is realistic to think of comparing a thousand drugs would be a pre-clinical screening program and the appropriate advice should relate to assay-related issues and cost-benefit analyses with consideration of what happens to the drugs that are classed as 'hits'. It is not a purely statistical issue.

If you perform a t-test (Welch or standard), or any other significance test (e.g. a permutations test), you will have a list of P-values that would allow you to rank the drugs on the basis of the statistical evidence in their respective datasets against their respective null hypotheses. That might be satisfactory for a statistics student, but it would be terrible for a clinician because it would allow a drug for which there is strong evidence for a minor clinical benefit to rank higher than a drug with moderate evidence for a huge benefit.

You need criteria for ranking that include the features that are important to the types of inference that you need to make. If you choose the mean benefit then you do not need any statistical procedure beyond calculation of the means.

Ranking drugs should be more complicated than you might think. For example, some drugs benefit only a subset of patients, and some cause harm to only a subset. Treating the patients as a single population might lead to a drug that is curative to some people ranking lower than a drug that brings a minor benefit to most patients. Another example of complexity is the fact that drugs may need different doses and regimens in different people.

Your specific questions:

  1. You could use an array of t-tests to compare each drug to every other drug but that would, as you imply, lead to a complicated task of ranking in multiple dimensions. Consider a system where each drug's effect is compared to an appropriate control dataset. You might use the same control data for each drug if it was necessary. That would give a one-dimensional array of p-values. (I would not attempt to rank drugs on the basis of just p-values!)

  2. The t-test you describe is a two-tailed version (i.e. non-directional). It treats positive and negative effect directions the same and so you would need to manually deal with the fact that a drug that makes the patients worse could rank higher than one that makes the patients better. Use a one-tailed version of the test so that the drugs that cause a benefit will have lower p-values than those that are detrimental.

  • $\begingroup$ Thanks @Michael Lew for the thoughtful response. The drug example was purely hypothetical. I was just wondering whether there existed a general methodology for this. By any chance, do you know of any references that have addressed a similar problem as mentioned above? It does not have to be a drug-related example. I'm just curious how they pursued the analysis in their context so that I can get ideas to implement in my example. Thanks! $\endgroup$
    – secondrate
    Commented Jan 5, 2022 at 22:47
  • $\begingroup$ There is no general methodology for deciding on criteria for 'best'. It always involves specific knowledge of the system and the analytical and inferential objectives. Maybe you need to ask a different question. $\endgroup$ Commented Jan 6, 2022 at 3:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.