2
$\begingroup$

I have a dataset that gives me the test results of 50 different patients. Each patient is tested 10 times. So I have 500 data points of results. A '1' in the matrix represents the test coming back positive, '0' if the test comes back negative. It is already known that all of the patients contain the disease. I am trying to quantify the accuracy of the testing method.

The problem is that I have 25 patients (half the population) that had only one result come back as positive. There are other patients that came back with all results as positive.

How do I calculate the accuracy of the test methodology? If I count the number of positive results, divided by the number of total results - I get a 50% accuracy rate. Intuitively, this does not seem correct given that I mentioned that 25 of the patients came back with a single positive result, even though we already know that they contain the disease. How do I include the inconsistency from patients that has only a single positive test result?

x = [[0,0,0,1,1,1,0,0,1,0],
     [1,0,0,0,0,0,0,0,1,0],
     ...
     [1,0,0,0,0,0,0,0,0,0],
     [0,1,0,0,0,0,0,0,0,0]]
$\endgroup$
1
  • $\begingroup$ Do you know for sure that all 50 patients were infected from the start of the study, so that a 'gold standard' test would have given 10 positive tests for every one of the 50? $\endgroup$
    – BruceET
    Jul 7, 2020 at 3:55

1 Answer 1

1
$\begingroup$

This Answer is exploratory. If you knew for sure from the beginning of your experiment that all 50 patients were infected, so that a 'gold standard' tests would have given 10 positive results for every one of the 50 patients, then I don't see how you could get the reported results as I understand them.

If all patients were infected from the start, essentially you are checking the sensitivity of the test, $P(\mathrm{Positive}|\mathrm{Disease}).$

If all patients are infected and sequential tests on a patient are independent evaluations, and if sensitivity is about 50%, then a dataset like yours can be simulated in R as follows:

set.seed(706)
n = 50;  r = 10
x = rbinom(n*r, 1, .5)             # outcomes of 500 tests
MAT = matrix(x, byrow=T, nrow=50)  # 50 x 10 matrix of results

Simulated results for first six patients:

head(MAT)
     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,]    0    0    0    1    0    1    0    1    1     0
[2,]    0    1    0    1    1    1    1    0    1     1
[3,]    0    1    1    0    1    0    1    0    0     1
[4,]    0    1    0    1    1    0    1    1    0     0
[5,]    0    0    1    0    0    0    0    1    1     1
[6,]    0    1    1    0    0    0    0    1    1     1

We have 50 estimates of sensitivity:

sens.est = rowMeans(MAT)
mean(sens.est)
[1] 0.518

Other descriptive statistics of the estimates are as follows:

summary(sens.est)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  0.100   0.400   0.500   0.518   0.600   0.900 

Each patient can give a sensitivity estimate of $0, .1, .2, \dots .9, 1.$ Here are the frequencies with which these estimates occur (in 50 patients).

table(sens.est)
sens.est
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 
  1   1   5  11  11  10   7   3   1 

Only one patient gave estimate 0.1 (with one positive result in 10). That's a long way from 25. In general, fewer than 1 patient in 100 would have just a single positive result.

dbinom(1, 10, .5)
[1] 0.009765625

So based on the assumptions I have made, there would be no way to account for 50% positive results overall and 25 of 50 subjects having just one positive result.

So now your job is to figure out which of my assumptions---or yours---is likely to be incorrect. If we can get the right model for your patients and your test, then maybe some progress is possible.

A good start might be to make a histogram of the 50 sensitivity estimates in your data. Do half of your patients tend to show mostly positive test results and the other half tend to show only one? Does your test detect only currently infected patients? Or should it it give positive results to anyone who has had the disease at any time before testing began?

In terms of numbers of positive tests, here is a histogram of my simulated data. It follows the model $\mathsf{Binom}(n=10,p=.5)$ (red dots) reasonably well for a sample of 50. [Values 0 and 10 are not impossible, each has probability $(1/2)^{10} \approx 0.001.]$ Did any of your patients happen to have $0$ or $10$ positive tests?

cutp=seq(-.5, 10.5, by=1)
hist(10*sens.est, prob=T, ylim=c(0,.3), br=cutp, col="skyblue2")
 k=0:10; pdf=dbinom(k, 10, .5)
 points(k, pdf, pch=10, col="red")

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.