Stats: Relationship between Alpha and Beta

Question

My question has to do with the relationship between alpha and beta and their definitions in statistics.

alpha = type I error rate = significance level under consideration that the NULL hypothesis is correct

Beta = type II error rate

If alpha is lowered (specificity increases as alpha = 1- specificity), beta increases (sensitivity/power decreases as beta = 1 - sensitivity/power)

How does a change in alpha affects beta ? Is there a linear relationship or not ? Do the ratio alpha/beta always the same, in other words the ratio specificity/sensitivity is always the same ? If yes, it means that by using a bonferroni correction we're just shifting to lower sensitivity and higher specificity but we're not changing sensitivity/specificity ratio. Is that correct to say so ?

Update (Case-specific question):

For a given experimental design, we run 5 Linear Models on the data. We have a True Positive Rate (sensitvity/power) at 0.8 and a True Negative Rate (specificity) at 0.7. (Let's imagine we know what should be positive and what should not.). If we now correct the significance level using Bonferroni to 0.05 / 5 = 0.01. Can we numerically estimate the resulting True Positive Rate (sensitivity/power) and True Negiative Rate (Specificity) ?

Thanks a lot for your help.

Answer 1

$\alpha$ and $\beta$ are related. I'll try to illustrate the point with a diagnostic test. Let's say that you have a diagnostic test that measures the level of a blood marker. It is known that people having a certain disease have lower levels of this marker compared to healthy people. It is immediately clear that you have to decide a cutoff value, below which a person is classified as "sick" whereas people with values above this cutoff are thought to be healthy. It is very likely, though, that the distribution of the bloodmarker varies considerably even within sick and healthy people. Some healthy persons might have very low blood marker levels, even though they are perfectly healthy. And some sick people have high levels of the blood marker even though they have the disease.

There are four possibilities that can occur:

1. a sick person is correctly identified as sick (true positive = TP)
2. a sick person is falsely classified as healthy (false negative = FN)
3. a healthy person is correctly identified as healthy (true negative = TN)
4. a healthy person is falsely classified as sick (false positive = FP)

These possibilities can be illustrated with a 2x2 table:

               Sick Healthy
Test positive   TP     FP
Test negative   FN     TN

$\alpha$ denotes the false positive rate, which is $\alpha = FP/(FP + TN)$. $\beta$ is the false negative rate, which is $\beta = FN/(TP + FN)$. I wrote a simply R script to illustrate the situation graphically.

alphabeta <- function(mean.sick=100, sd.sick=10, 
    mean.healthy=130, sd.healthy=10, cutoff=120, n=10000, 
    side="below", do.plot=TRUE) {
      
      popsick <- rnorm(n, mean=mean.sick, sd=sd.sick)
      pophealthy <- rnorm(n, mean=mean.healthy, sd=sd.healthy)
      
      if ( side == "below" ) {
        
        truepos <- length(popsick[popsick <= cutoff])
        falsepos <- length(pophealthy[pophealthy <= cutoff])
        trueneg <- length(pophealthy[pophealthy > cutoff])
        falseneg <- length(popsick[popsick > cutoff])
        
      } else if ( side == "above" ) {
        
        truepos <- length(popsick[popsick >= cutoff])
        falsepos <- length(pophealthy[pophealthy >= cutoff])
        trueneg <- length(pophealthy[pophealthy < cutoff])
        falseneg <- length(popsick[popsick < cutoff])
        
      }
      
      twotable <- matrix(c(truepos, falsepos, falseneg, trueneg), 
                    2, 2, byrow=T)
      rownames(twotable) <- c("Test positive", "Test negative")
      colnames(twotable) <- c("Sick", "Healthy")
      
      spec <- twotable[2, 2]/(twotable[2, 2] + twotable[1, 2])
      alpha <- 1 - spec
      sens <- pow <- twotable[1, 1]/(twotable[1, 1] + 
                      twotable[2, 1])
      beta <- 1 - sens
      
      pos.pred <- twotable[1,1]/(twotable[1,1] + twotable[1,2])
      neg.pred <- twotable[2,2]/(twotable[2,2] + twotable[2,1])
      
      if ( do.plot == TRUE ) {        
        dsick <- density(popsick)
        dhealthy <- density(pophealthy)
        
        par(mar=c(5.5, 4, 0.5, 0.5))
        plot(range(c(dsick$x, dhealthy$x)), range(c(c(dsick$y, 
       dhealthy$y))), type = "n", xlab="", ylab="", 
           axes=FALSE)
        box()
        axis(1, at=mean(pophealthy), 
         lab=substitute(mu[H[0]]~paste("=", m, sep=""), 
         list(m=mean.healthy)), cex.axis=1.5, tck=0.02)
        axis(1, at=mean(popsick), 
           lab=substitute(mu[H[1]] ~ paste("=", m, sep=""), 
           list(m=mean.sick)), cex.axis=1.5, tck=0.02)                                        
        axis(1, at=cutoff, lab=substitute(italic(paste("Cutoff=", 
         coff, sep="")), list(coff=cutoff)), pos=-0.004, 
          tick=FALSE, cex.axis=1.25)
        lines(dhealthy, col = "steelblue", lwd=2)
        
        if ( side == "below" ) {
          polygon(c(cutoff, dhealthy$x[dhealthy$x<=cutoff], 
          cutoff), c(0, dhealthy$y[dhealthy$x<=cutoff],0), 
            col = "grey65")
        } else if ( side == "above" ) {
          polygon(c(cutoff, dhealthy$x[dhealthy$x>=cutoff], 
            cutoff), c(0, dhealthy$y[dhealthy$x>=cutoff],0), 
            col = "grey65")
        }
        
        lines(dsick, col = "red", lwd=2)
        
        if ( side == "below" ) {
          polygon(c(cutoff, dsick$x[dsick$x>cutoff], cutoff), 
          c(0, dsick$y[dsick$x>cutoff], 0) , col="grey90")
        } else if ( side == "above" ) {
          polygon(c(cutoff, dsick$x[dsick$x<=cutoff], cutoff), 
            c(0, dsick$y[dsick$x<=cutoff],0) , col="grey90")
        }
        
        legend("topleft",
               legend= 
          (c(as.expression(substitute(alpha~paste("=", a), 
           list(a=round(alpha,3)))), 
             as.expression(substitute(beta~paste("=", b), 
             list(b=round(beta,3)))))), fill=c("grey65", 
             "grey90"), cex=1.2, bty="n")
        abline(v=mean(popsick), lty=3)
        abline(v=mean(pophealthy), lty=3)
        abline(v=cutoff, lty=1, lwd=1.5)
        abline(h=0)
        
      }
      
      #list(specificity=spec, sensitivity=sens, alpha=alpha, beta=beta, power=pow, positiv.predictive=pos.pred, negative.predictive=neg.pred)
      
      c(alpha, beta)
      
    }

Let's look at an example. We assume that the mean level of the blood marker among the sick people is 100 with a standard deviation of 10. Among the healthy people, the mean blood level is 140 with a standard deviation of 15. The clinician sets the cutoff at 120.

    alphabeta(mean.sick=100, sd.sick=10, mean.healthy=140, 
         sd.healthy=15, cutoff=120, n=100000, do.plot=TRUE, 
         side="below")

                  Sick Healthy
    Test positive 9764     901
    Test negative  236    9099

You see that the shaded areas are in a relation with each other. In this case, $\alpha = 901/(901+ 9099) \approx 0.09$ and $\beta = 236/(236 + 9764)\approx 0.024$. But what happens if the clinician had set the cutoff differently? Let's set it a bit lower, to 105 and see what happens.

              Sick Healthy
Test positive 6909      90
Test negative 3091    9910

Our $\alpha$ is very low now because almost no healthy people are diagnosed as sick. But our $\beta$ has increased, because sick people with a high blood marker level are now falsely classified as healthy.

Finally, let us look how $\alpha$ and $\beta$ change for different cutoffs:

    cutoffs <- seq(0, 200, by=0.1)
    cutoff.grid <- expand.grid(cutoffs)
    
    plot.frame <- apply(cutoff.grid, MARGIN=1, FUN=alphabeta, 
        mean.sick=100, sd.sick=10, mean.healthy=140, 
        sd.healthy=15, n=100000, do.plot=FALSE, side="below")
    
    plot(plot.frame[1,] ~ cutoffs, type="l", las=1, 
      xlab="Cutoff value", ylab="Alpha/Beta", lwd=2, 
      cex.axis=1.5, cex.lab=1.2)
    lines(plot.frame[2,]~cutoffs, col="steelblue", lty=2, lwd=2)
    legend("topleft", legend=c(expression(alpha), 
       expression(beta)), lwd=c(2,2),lty=c(1,2), col=c("black", 
       "steelblue"), bty="n", cex=1.2)

You can immediately see that the ratio of $\alpha$ and $\beta$ is not constant. What is also very important is the effect size. In this case, this would be the difference of the means of the blood marker levels among sick and healthy people. The greater the difference, the easier the two groups can be separated by a cutoff:

Here we have a "perfect" test in the sense that the cutoff of 150 discriminates the sick from the healthy.

---

Bonferroni adjustements

Bonferroni adjustments reduces the $\alpha$ error but inflate the type II error ($\beta$). This means that the error of making a false negative decision is increased while false positives are minimized. That's why the Bonferroni adjustment is often called conservative. In the graphs above, note how the $\beta$ increased when we lowered the cutoff from 120 to 105: it increased from $0.02$ to $0.31$. At the same time, $\alpha$ decreased from $0.09$ to $0.01$.

Answer 2

For others in the future:

In Sample Size estimation, the Ztotal is calculated by adding the Z corresponding to alpha and Z corresponding to power (1-beta). So mathematically, if sample size is kept constant, increasing Z for alpha means you decrease the Z for power by the SAME amount e.g., increasing Zalpha from 0.05 to 0.1 decreases Zpower by 0.05.

The difference is the Z for alpha is two-tailed while the Z for beta is 1-tailed. So, while the Z value changes by the same amount, but the probability % that this Z value corresponds to does not change by the same amount.

Example:

5% alpha (95% confidence) with 80% power (20% beta) gives the same sample size as

20% alpha (80% confidence) with 93.6% power (6.4% beta) rather than the 95% power we would have if the relationship were 1:1.

Answer 3

There is no general relation between alpha and beta.

It all depends on your test, take the simple exemple:

(Wikipedia)

In colloquial usage type I error can be thought of as "convicting an innocent person" and type II error "letting a guilty person go free".

A jury can be severe: no type II error, some type I A jury can be "kind": no type I but some type II A jury can be normal: some type I and some type II A jury can be perfect: no error

In practice there is two antagonist effect:

When the quality of the test goes up, type I and type II error decrease until some point. When a jury improves, he tends to give better judgment over both innocent and guilty people.

After some point the underlying problem appears in the building of the test. Type I or II are more important for the one who runs the test. With the jury exemple, type I errors are more important and so the law process is build to avoid type I. If there is any doubt the person is free. Intuitively this lead to a growth in type II error.

Concerning Bonferroni:

(Wikipedia again)

Bonferroni correction controls the probability of false positives only. The correction ordinarily comes at the cost of increasing the probability of producing false negatives, and consequently reducing statistical power. When testing a large number of hypotheses, this can result in large critical values.