We have a potential biomarker for predicting whether a patient has cancer or not. The biomarker test result is binary being positive or negative. We want to get some sense of the amount of patients that need to be tested to determine whether this biomarker is a good predictor or not.

From reading on the internet it seems that the way to go is to look at the sensitivity (for the number of cases) and specificity (for the number of controls). It is suggested that you should treat this situation as a one-sample proportion test, but it remains unclear how you should go about estimating what the sensitivity is and the range you are prepared to except. If say I consider any biomarker with a sensitivity of greater than 0.8 to be "good", how would you set the two variables up? I would like my null hypothesis to be the biomarker is no better than a random assignment i.e. a sensitivity of 0.5. Could anyone give an example of the best way to do this (especially if it is in R).

  • $\begingroup$ Are you saying that you are going to start with a set of known cases, next perform your biomarker test (collect data), and estimate the sensitivity? And you will start with a set of known controls, collect data, and estimate specificity? $\endgroup$ – user1108 Jan 25 '12 at 22:38
  • $\begingroup$ For this calculation in effect yes. In reality we will not know prior to patient recruitment, but will keep recruiting until we have enough cases and controls. Also we have an estimated rate that a patient will be a case so we can use that to estimate the total number we will need to recruit, $\endgroup$ – danielsbrewer Jan 26 '12 at 11:24
  • $\begingroup$ If the biomarker only gives a yes/no-answer then you might go with sensitivity/specificity and do planning in a context for tests for proportions. If a value for one of them is "good" or "bad" depends on the real life consequences of a false decision. If the biomarker originally gives a continuous measurement then ROC-curves and AUC-statistics and corresponding sample size planning methods may be more appropriate. But all this only scratches the surface of the methods connected with diagnostic testing... $\endgroup$ – psj Jan 27 '12 at 13:08

Let's talk about sensitivity (which we'll denote by $p$), the specificity is similar. The following is a frequentist approach; it would be great if one of the Bayesians here could add another answer to discuss an alternative way to go about it.

Suppose you've recruited $n$ people with cancer. You apply your biomarker test to each, so you will get a sequence of 0's and 1's which we'll call x. The entries of x will have a Bernoulli distribution with success probability $p$. The estimate of $p$ is $\hat{p} = \sum x /n$. Hopefully $\hat{p}$ is "big", and you can judge the precision of your estimate via a confidence interval for $p$.

Your question says that you'd like to know how big $n$ should be. To answer it you'll need to consult the biomarker literature to decide how big is "big" and how low of a sensitivity you can tolerate due to sampling error. Suppose you decide that a biomarker is "good" if its sensitivity is bigger than $p = 0.5$ (that's actually not so good), and you'd like $n$ to be big enough so there's a 90% chance to detect a sensitivity of $p = 0.57$. Suppose you'd like to control your significance level at $\alpha = 0.05$.

There are at least two approaches - analytical and simulation. The pwr package in R already exists to help with this design - you need to install it first. Next you'll need an effect size, then the function you want is pwr.p.test.

h1 <- ES.h(0.57, 0.5)
pwr.p.test(h = h1, n = NULL, sig.level = 0.05, power = 0.9, alt = "greater")

     proportion power calculation for binomial distribution (arc... 

              h = 0.1404614
              n = 434.0651
      sig.level = 0.05
          power = 0.9
    alternative = greater

So you'd need around $435$ people with cancer to detect a sensitivity of $0.57$ with power $0.90$ when your significance level is $0.05$. I've tried the simulation approach, too, and it gives a similar answer. Of course, if the true sensitivity is higher than $0.57$ (your biomarker is better) then you'd need fewer people to detect it.

Once you've got your data, the way to run the test is (I'll simulate data for the sake of argument).

n <- 435
sens <- 0.57
x <- rbinom(n, size = 1, prob = sens)
binom.test(sum(x), n, p = 0.5, alt = "greater")

    Exact binomial test

data:  sum(x) and n 
number of successes = 247, number of trials = 435,
p-value = 0.002681
alternative hypothesis: true probability of success is greater than 0.5 
95 percent confidence interval:
 0.527342 1.000000 
sample estimates:
probability of success 

The estimate of sensitivity is $0.568$. What really matters is the confidence interval for $p$ which in this case is $[0.527, 1]$.

EDIT: If you like the simulation approach better, then you can do it this way: set

n <- 435
sens <- 0.57
nSim <- 1000

and let runTest be

runTest <- function(){
  x <- rbinom(1, size = n, prob = sens)
  tmp <- binom.test(x, n, p = 0.5, alt = "greater")
  tmp$p.value < 0.05

so the estimate of power is

mean(replicate(nSim, runTest()))
[1] 0.887

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.