Test a single score against a sampled distribution of scores

Question

I am currently working on a tool which calculates a score for each sample (i.e. patient biopsy) it is run on. Input samples can belong to one of two categories: case and control. Usually, a user would run the tool on a number of samples from both categories. For each case sample, I would like to report whether the sample's score is significantly different from the scores of the provided control samples. In other words, for each case sample I want to assess the likelyhood of seeing a score at least as extreme as the observed score if the sample came from the control distribution. My problem is that I don't know the real control distribution; I can only estimate it based on the control samples the user provides.

Currently, I am calculating the mean and standard deviation of the scores of the control group. For every case sample, I am then calculating the z-score with respect to the control group (i.e. I subtract the control group's mean score from the sample's score and divide by the control group's standard deviation). If the absolute value of the sample's zscore is > 2, I report the sample as significantly different. However, I a worried that with this approach I do not take the sample size of the control group into account. Other simple approaches like t-tests seem not appropriate here, since I don't want to compare means.

Could someone suggest a more appropriate test for this use-case? Any help would be greatly appreciated.

Thank you,

Peter

Answer 1

You may want to use a prediction interval when considering a new patient, who was not part of the previous analysis for the control group.

Suppose the true scores for the control group are distributed $\mathsf{Norm}(\mu =50, \sigma=7).$ In a real application you could not know the population mean and standard deviation, but would estimate them using a sample mean $\bar X = 48.90$ and standard deviation $S = 7.03$ of available data. (Computations in R:)

set.seed(1234)
x = rnorm(100, 50, 7)
mean(x);  sd(x)
[1] 48.90267
[1] 7.030837

Confidence interval. Then you might use these estimates to compute a 95% confidence interval for $\mu,$ of the form $$\bar X \pm t^*S\sqrt{\frac{1}{n}},$$ where $t^* = 1.98$ cuts probability 2.5% from the upper tail of Student's t distribution with $\nu = n-1 = 99$ degrees of freedom.

qt(.975, 99)
[1] 1.984217

Based on the estimates above, the 95% CI computes to $(47.51, 50.30).$

CI = mean(x) + qt(c(.025,.975), 99)*sd(x)/sqrt(100); CI
[1] 47.50760 50.29774

Prediction interval. Now suppose you have a new patient with score $55.7.$ Your first inclination might be to say this patient is not contained in the 95% CI above and so would not be consistent with the control group. However, the CI is meant as an estimate of the unknown population mean $\mu.$ It is not intended to 'predict' scores of individual additional cancer-free patients.

A 95% prediction interval is of the form $$\bar X \pm t^*S\sqrt{1+\frac{1}{n}},$$ which takes into account the variability of the new patient. This prediction interval $(34.88, 62.92)$ does include the new patient's score.

PI = mean(x) + qt(c(.025,.975), 99)*sd(x)*sqrt(1+1/100); PI
[1] 34.88238 62.92295

Notes: Without seeing your real data, I should not take this further.

(1) In an actual situation, you would hope that prediction intervals for control and treatment groups do not overlap, so we can be reasonably sure how to classify the new patient.

(2) If the only purpose of prediction intervals is to classify new patients, then it might be appropriate to use one-sided prediction intervals, with an upper bound for control group and a lower bound for treatment group.

(3) Some sort of discriminant analysis might take various kinds of information on a new patient into account in deciding on a classification.