# Test a single score against a sampled distribution of scores

I am currently working on a tool which calculates a score for each sample (i.e. patient biopsy) it is run on. Input samples can belong to one of two categories: case and control. Usually, a user would run the tool on a number of samples from both categories. For each case sample, I would like to report whether the sample's score is significantly different from the scores of the provided control samples. In other words, for each case sample I want to assess the likelyhood of seeing a score at least as extreme as the observed score if the sample came from the control distribution. My problem is that I don't know the real control distribution; I can only estimate it based on the control samples the user provides.

Currently, I am calculating the mean and standard deviation of the scores of the control group. For every case sample, I am then calculating the z-score with respect to the control group (i.e. I subtract the control group's mean score from the sample's score and divide by the control group's standard deviation). If the absolute value of the sample's zscore is > 2, I report the sample as significantly different. However, I a worried that with this approach I do not take the sample size of the control group into account. Other simple approaches like t-tests seem not appropriate here, since I don't want to compare means.

Could someone suggest a more appropriate test for this use-case? Any help would be greatly appreciated.

Thank you,

Peter

• Trying to understand... when you say "I am currently working on a tool which calculates a score for each sample": how this score is calculated? Commented Sep 20, 2021 at 13:55
• @Pitouille : thanks for trying to help! The tool uses a complex procedure (explaining would go too far here) to summarize epigenetic properties of the patient sample into a single, real valued number (can be positive or negative). The idea would be that cancer patients would often get a higher score than the control samples, due to their epigenetic properties. It may be reasonable to assume the scores of the controls are roughly normally distributed. Does that help? Commented Sep 20, 2021 at 15:01

You may want to use a prediction interval when considering a new patient, who was not part of the previous analysis for the control group.

Suppose the true scores for the control group are distributed $$\mathsf{Norm}(\mu =50, \sigma=7).$$ In a real application you could not know the population mean and standard deviation, but would estimate them using a sample mean $$\bar X = 48.90$$ and standard deviation $$S = 7.03$$ of available data. (Computations in R:)

set.seed(1234)
x = rnorm(100, 50, 7)
mean(x);  sd(x)
[1] 48.90267
[1] 7.030837


Confidence interval. Then you might use these estimates to compute a 95% confidence interval for $$\mu,$$ of the form $$\bar X \pm t^*S\sqrt{\frac{1}{n}},$$ where $$t^* = 1.98$$ cuts probability 2.5% from the upper tail of Student's t distribution with $$\nu = n-1 = 99$$ degrees of freedom.

qt(.975, 99)
[1] 1.984217


Based on the estimates above, the 95% CI computes to $$(47.51, 50.30).$$

CI = mean(x) + qt(c(.025,.975), 99)*sd(x)/sqrt(100); CI
[1] 47.50760 50.29774


Prediction interval. Now suppose you have a new patient with score $$55.7.$$ Your first inclination might be to say this patient is not contained in the 95% CI above and so would not be consistent with the control group. However, the CI is meant as an estimate of the unknown population mean $$\mu.$$ It is not intended to 'predict' scores of individual additional cancer-free patients.

A 95% prediction interval is of the form $$\bar X \pm t^*S\sqrt{1+\frac{1}{n}},$$ which takes into account the variability of the new patient. This prediction interval $$(34.88, 62.92)$$ does include the new patient's score.

PI = mean(x) + qt(c(.025,.975), 99)*sd(x)*sqrt(1+1/100); PI
[1] 34.88238 62.92295


Notes: Without seeing your real data, I should not take this further.

(1) In an actual situation, you would hope that prediction intervals for control and treatment groups do not overlap, so we can be reasonably sure how to classify the new patient.

(2) If the only purpose of prediction intervals is to classify new patients, then it might be appropriate to use one-sided prediction intervals, with an upper bound for control group and a lower bound for treatment group.

(3) Some sort of discriminant analysis might take various kinds of information on a new patient into account in deciding on a classification.

• Thank you, that was very helpful! I especially appreciate the examples. I expect that the variability in the cancer group to be quite high, but the variability for the control group should be much lower. Based on your suggestions, I would now calculate the prediction interval for the control group, and mark each cancer sample as "significantly different" if its score is outside the control-group's prediction interval. I'd rather avoid treating the cancer samples as a unified group, so I wouldn't calculate a cancer-group prediction interval. Commented Sep 21, 2021 at 7:39
• Seems reasonable. Commented Sep 21, 2021 at 7:42
• Thanks again. I just want to point out a typo in the above command, in case someone else wants to use it: As far as I understood, the R command for calculating the pi is missing a "sqrt()", and it should be pi = mean(x) + qt(c(.025,.975), 99)*sd(x)*sqrt(1+1/100); pi Commented Sep 21, 2021 at 12:18
• @peter_pen. Thanks much for finding typo. Fixed it // For the record, another blunder (also fixed) was to use pi which is a reserved constant in R for 3.141593... and should never be used for anything else. Because R is case-sensitive, it's OK to use PI. Commented Sep 22, 2021 at 17:25