How do I find a variance-stabilizing transformation?

Question

I wonder how to solve this classical problem:

Recall that for a binomial proportion $\hat p$ based on a sample of size $n$ we have $$E(\hat p)=p$$ and $$\operatorname{Var}(\hat p) = p(1-p)/n.$$ Show that the variance-stabilizing transformation of $\hat p$ is $2\sqrt{n}\sin^{-1}\sqrt{\hat p}$. This is called the arcsin square-root transformation.

What do I have to integrate? I'm a bit confused about the notation, so please, could anybody give me some advice?

Answer 1

Since the question partly concerns notation and basic concepts, I will be expansive in the following answer, making sure to motivate, describe, and explain the notation, the statistical reasoning, and the mathematical steps used. I hope this has been done in a way that clearly indicates how similar problems can be solved.

---

"Sample of size $n$" means you have $n$ independent and identically distributed binary observations, often called "success" and "failure". The underlying distribution is determined by the chance of success, which is a number $p$. An estimate of $p$ based on the sample is the proportion of successes, written $\hat p$. It is a random variable.

Because $\hat p$ is bounded (between $0$ and $1$), it has a mean $\mathbb{E}(\hat p)$ and a variance $\operatorname{Var}(\hat p)$. These can be figured out in terms of the underlying chance of success $p$; they are

$$\mathbb{E}(\hat p)=p$$

and

$$\operatorname{Var}(\hat p) = \frac{p(1-p)}{n}.\tag{1}$$

A variance-stabilizing transformation is a function $f$ that converts all possible values of $\hat p$ into other values $Y=f(\hat p)$ in such a way that the variance of $Y$ is constant--usually taken to be $1$. This makes $Y$ simpler to work with because (at least insofar as its first two moments are concerned) it is characterized by its expectation alone, rather than by an expectation and a variance.

It's easy to change the variance of any random variable $X$: multiplying $X$ by a constant, say $\lambda$, multiplies its variance by $\lambda^2$. Dividing $X$ by the square root of its variance will thereby give it a unit variance. We would therefore like the effect of $f$ to be that of dividing $\hat p$ by the square root of $1/\operatorname{Var}(\hat p)$, no matter what value $\hat p$ might take on.

At this point it is difficult to proceed because we don't actually know $\operatorname{Var}(\hat p)$. However, we can hope the estimate $\hat p$ is close to $p$, in which case we could approximate $\operatorname{Var}(\hat p)$ by plugging $\hat p$ in place of $p$ in formula $(1)$:

$$\widehat{\operatorname{Var}}(\hat p) = \frac{\hat p(1-\hat p)}{n}.\tag{2}$$

This is enough information to find a differentiable transformation $f$. Recall that "differentiable" means $f$ has a well-defined slope at any argument $x$ (with $0\lt x \lt 1$ in this situation). The slope, written $f^\prime(x)$, can be considered a local scaling factor: it is the amount by which $f$ rescales values close to $x$. One can write $$df(x) = f^\prime(x)dx$$ where $dx$ is a small change in $x$ and $df(x)$ is the corresponding change in $f(x)$.

Above, we have seen this local scaling factor ought to be the reciprocal square root of the variance. Using the estimate of that variance in $(2)$ enables us to write the desired property of $f$ in the form

$$df(\hat p) = f^\prime(\hat p)d\hat p = \sqrt{\frac{1}{\widehat{\operatorname{Var}(\hat p)}}}d\hat p= \sqrt{\frac{n}{\hat p(1-\hat p)}}d\hat p.\tag{3}$$

The differential $d\hat p$ is the scale factor near $\hat p$ while the differential $df(\hat p)$ is the scale factor near $f(\hat p)$: as promised, they are related through multiplication by $f^\prime(\hat p)$.

The Fundamental Theorem of Calculus asserts that such first order differential equations for an unknown function $f$ are solved via integration. In this case a solution is readily found by squinting hard at the right hand side of $(3)$ in a search for similar mathematical patterns. The square root suggests we would love for both $\hat p$ and $1-\hat p$ themselves to be squares. In particular, if $\hat p = s^2$ and $1-\hat p = c^2$, then

$$\hat p(1-\hat p) = s^2(1-s^2) = s^2(c^2).$$

That ought to remind anyone of the trigonometric functions sine and cosine. Indeed, for angles $0 \le \theta \le \pi/2$, both the sine and cosine range between $0$ and $1$, implying we really can write $\hat p(\theta) = \sin^2\theta$ for some angle $\theta$. This permits us to write

$$df(\theta) = df(\hat p(\theta)) = f^\prime(\sin^2\theta)d\hat p = \sqrt{\frac{n}{\sin^2\theta\cos^2\theta}}\left(2\sin\theta\cos\theta\right)d\theta = 2\sqrt{n}d\theta.\tag{4}$$

To achieve this simple form I have performed the only real calculation in this answer: the differential of $\hat p = \sin^2\theta$ is $d\hat p=2\sin\theta\cos\theta d\theta$, as asserted by the Chain Rule.

At this point we are done, because when two differential expressions defined on a connected set (like the interval $[0,1]$) are equal, as in the two sides of equation $(4)$, the functions of which they are differentials differ only by an additive constant--but such additive changes of random variables will not alter their variances, so this doesn't matter. Consequently we may take

$$Y = 2\sqrt{n}\theta.$$

Since $\hat p = \sin^2\theta$, we may solve for $\theta=\arcsin\sqrt{\hat p}$ in terms of $\hat p$, giving one solution

$$f(\hat p) = Y = 2\sqrt{n} \arcsin\sqrt{\hat p}.$$

All other solutions are related to that one by some combination of adding a constant (which won't change the variance, as we have noted) and multiplying by some constant (which, although it changes the variance, still preserves the property of being constant).

---

According to this analysis, $f$ deserves only to be called an approximate variance-stabilizing transformation, rather than the variance-stabilizing transformation. It nevertheless will perform well when $\hat p$ has a reasonably high chance of being close to $p$ itself. This typically is the case when both $n\hat p$ and $n(1-\hat p)$ both exceed $5$--but the threshold $5$ can be modified to suit your own criteria for "reasonably high" and "close to."