# How do I find a variance-stabilizing transformation?

I wonder how to solve this classical problem:

Recall that for a binomial proportion $\hat p$ based on a sample of size $n$ we have $$E(\hat p)=p$$ and $$\operatorname{Var}(\hat p) = p(1-p)/n.$$ Show that the variance-stabilizing transformation of $\hat p$ is $2\sqrt{n}\sin^{-1}\sqrt{\hat p}$. This is called the arcsin square-root transformation.

What do I have to integrate? I'm a bit confused about the notation, so please, could anybody give me some advice?

Since the question partly concerns notation and basic concepts, I will be expansive in the following answer, making sure to motivate, describe, and explain the notation, the statistical reasoning, and the mathematical steps used. I hope this has been done in a way that clearly indicates how similar problems can be solved.

"Sample of size $$n$$" means you have $$n$$ independent and identically distributed binary observations, often called "success" and "failure". The underlying distribution is determined by the chance of success, which is a number $$p$$. An estimate of $$p$$ based on the sample is the proportion of successes, written $$\hat p$$. It is a random variable.

Because $$\hat p$$ is bounded (between $$0$$ and $$1$$), it has a mean $$\mathbb{E}(\hat p)$$ and a variance $$\operatorname{Var}(\hat p)$$. These can be figured out in terms of the underlying chance of success $$p$$; they are

$$\mathbb{E}(\hat p)=p$$

and

$$\operatorname{Var}(\hat p) = \frac{p(1-p)}{n}.\tag{1}$$

A variance-stabilizing transformation is a function $$f$$ that converts all possible values of $$\hat p$$ into other values $$Y=f(\hat p)$$ in such a way that the variance of $$Y$$ is constant--usually taken to be $$1$$. This makes $$Y$$ simpler to work with because (at least insofar as its first two moments are concerned) it is characterized by its expectation alone, rather than by an expectation and a variance.

It's easy to change the variance of any random variable $$X$$: multiplying $$X$$ by a constant, say $$\lambda$$, multiplies its variance by $$\lambda^2$$. Dividing $$X$$ by the square root of its variance will thereby give it a unit variance. We would therefore like the effect of $$f$$ to be that of dividing $$\hat p$$ by the square root of $$1/\operatorname{Var}(\hat p)$$, no matter what value $$\hat p$$ might take on.

At this point it is difficult to proceed because we don't actually know $$\operatorname{Var}(\hat p)$$. However, we can hope the estimate $$\hat p$$ is close to $$p$$, in which case we could approximate $$\operatorname{Var}(\hat p)$$ by plugging $$\hat p$$ in place of $$p$$ in formula $$(1)$$:

$$\widehat{\operatorname{Var}}(\hat p) = \frac{\hat p(1-\hat p)}{n}.\tag{2}$$

This is enough information to find a differentiable transformation $$f$$. Recall that "differentiable" means $$f$$ has a well-defined slope at any argument $$x$$ (with $$0\lt x \lt 1$$ in this situation). The slope, written $$f^\prime(x)$$, can be considered a local scaling factor: it is the amount by which $$f$$ rescales values close to $$x$$. One can write $$df(x) = f^\prime(x)dx$$ where $$dx$$ is a small change in $$x$$ and $$df(x)$$ is the corresponding change in $$f(x)$$.

Above, we have seen this local scaling factor ought to be the reciprocal square root of the variance. Using the estimate of that variance in $$(2)$$ enables us to write the desired property of $$f$$ in the form

$$df(\hat p) = f^\prime(\hat p)d\hat p = \sqrt{\frac{1}{\widehat{\operatorname{Var}(\hat p)}}}d\hat p= \sqrt{\frac{n}{\hat p(1-\hat p)}}d\hat p.\tag{3}$$

The differential $$d\hat p$$ is the scale factor near $$\hat p$$ while the differential $$df(\hat p)$$ is the scale factor near $$f(\hat p)$$: as promised, they are related through multiplication by $$f^\prime(\hat p)$$.

The Fundamental Theorem of Calculus asserts that such first order differential equations for an unknown function $$f$$ are solved via integration. In this case a solution is readily found by squinting hard at the right hand side of $$(3)$$ in a search for similar mathematical patterns. The square root suggests we would love for both $$\hat p$$ and $$1-\hat p$$ themselves to be squares. In particular, if $$\hat p = s^2$$ and $$1-\hat p = c^2$$, then

$$\hat p(1-\hat p) = s^2(1-s^2) = s^2(c^2).$$

That ought to remind anyone of the trigonometric functions sine and cosine. Indeed, for angles $$0 \le \theta \le \pi/2$$, both the sine and cosine range between $$0$$ and $$1$$, implying we really can write $$\hat p(\theta) = \sin^2\theta$$ for some angle $$\theta$$. This permits us to write

$$df(\theta) = df(\hat p(\theta)) = f^\prime(\sin^2\theta)d\hat p = \sqrt{\frac{n}{\sin^2\theta\cos^2\theta}}\left(2\sin\theta\cos\theta\right)d\theta = 2\sqrt{n}d\theta.\tag{4}$$

To achieve this simple form I have performed the only real calculation in this answer: the differential of $$\hat p = \sin^2\theta$$ is $$d\hat p=2\sin\theta\cos\theta d\theta$$, as asserted by the Chain Rule.

At this point we are done, because when two differential expressions defined on a connected set (like the interval $$[0,1]$$) are equal, as in the two sides of equation $$(4)$$, the functions of which they are differentials differ only by an additive constant--but such additive changes of random variables will not alter their variances, so this doesn't matter. Consequently we may take

$$Y = 2\sqrt{n}\theta.$$

Since $$\hat p = \sin^2\theta$$, we may solve for $$\theta=\arcsin\sqrt{\hat p}$$ in terms of $$\hat p$$, giving one solution

$$f(\hat p) = Y = 2\sqrt{n} \arcsin\sqrt{\hat p}.$$

All other solutions are related to that one by some combination of adding a constant (which won't change the variance, as we have noted) and multiplying by some constant (which, although it changes the variance, still preserves the property of being constant).

According to this analysis, $$f$$ deserves only to be called an approximate variance-stabilizing transformation, rather than the variance-stabilizing transformation. It nevertheless will perform well when $$\hat p$$ has a reasonably high chance of being close to $$p$$ itself. This typically is the case when both $$n\hat p$$ and $$n(1-\hat p)$$ both exceed $$5$$--but the threshold $$5$$ can be modified to suit your own criteria for "reasonably high" and "close to."