2
$\begingroup$

For a method, I'm calculating it's sensitivity and specificity. I also want to calculate standard errors, but I'm unsure how. I don't have a sample to calculate it from. All I have are 50 iterations where I applied the method to a different dataset each time. So that, I have 50 values for the number of TPs, FPs, TNs, FNs. From this, I calculate the sensitivity and specificity by summing all TPs, FPs, TNs, FNs.

I haven't been able to find much information online, aside from a wikihow article, which states that it can be calculated as:

$\sqrt{\frac{(1-Sensitivity)*Sensitivity}{n_p}} * 1.96,$

where $n_p$ is the number of positives in the dataset. Is this correct?

$\endgroup$
2

1 Answer 1

3
$\begingroup$

Let's define some notation.

$y\in\{-1,+1\}$ is the true state.

$\hat y\in\{-1,+1\}$ is the predicted state.

Then...

$$ \text{Sensitivity} = \mathbb{P}\big(\hat y = +1 \vert y = +1)\\ \text{Specificity} = \mathbb{P}\big(\hat y = -1 \vert y = -1) $$

So both of these are just proportions. Treat them like you would any other proportion parameter of a Bernoulli distribution. That's where you get the standard error formula you gave. The common way to calculate standard error (SD) is the following: The outcome of each experiment is Bernoulli distributed with parameter $p$ $X_1,\dots,X_n \overset{iid}{\sim} Bern(p)$. The number of successes is then binomially distributed (since order does not matter) and we need to deal with the https://en.m.wikipedia.org/wiki/Binomial_proportion_confidence_interval The standard error for the proportion is then $SD(\hat{p})=\sqrt{\dfrac{\hat{p}(1-\hat{p})}{n}}$. Other possible ways to compute a confidence interval are highlighted in Wikipedia.

(I dispute that the $1.96$ should be included. Standard errors should not be tied to particular confidence levels. What if I want a $90\%$ confidence interval or a $99\%$ confidence interval?))

To get from there to your formula, the $\hat{p}$ is the sensitivity, and the $n$ is the number of positive cases, which you are calling $n_p$. Likewise, the standard error for specificity would use $\hat{p} = \text{specificity}$ and $n$ equal to the number of true negative cases.

$\endgroup$
1
  • $\begingroup$ I don’t see a nice way to make the prose flow, but $\hat p$ is the estimate of $p$, calculated as the proportion of $1$s in the data. $\endgroup$
    – Dave
    Aug 28, 2021 at 4:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.