How to interpret the Delta Method?

I'm reading through https://www.statlect.com/asymptotic-theory/delta-method it defined the Delta Method as:

The delta method is a method that allows us to derive, under appropriate conditions, the asymptotic distribution of $g(\hat{\theta}_n)$ from the asymptotic distribution of $\hat{\theta}$.

and one example says, in short:

A sequence of $\hat{\theta}_i$ is asymptotically normal with mean=1 and variance=1. We want to derive the asymptotic distribution of the sequence $\hat{\theta}^2$

And the solution is:

$$\sqrt{n}(\hat{\theta}_n^2-1) \xrightarrow{D} N(0,4)$$

1. How do I interpret this result? This doesn't tell me the distribution of $\hat{\theta}_n^2$, instead it tells me the distribution of a shifted and scaled version of it.
2. The steps to arrive at the solution suggest the variance of $\hat{\theta}_n^2$ is 4, and they just plugged it into the $N(0,4)$ above. If this is true, how come the variance of $\hat{\theta}_n^2$ is the variance of $\sqrt{n}(\hat{\theta}_n^2-1)$ ?
• Are you dealing with the scalar case or the vector case? Are you comfortable with matrices and vectors? or do you want to keep it simple? – Matthew Gunn Nov 1 '16 at 7:29
• The idea here is that $g(x) = x^2$ – Matthew Gunn Nov 1 '16 at 7:39
• That first quoted sentence is a description of what it allows us to do rather than a definition of what it is. – Glen_b Nov 6 '16 at 3:41
• I think, there was a typo in the first quoted sentence. It should be $\hat{\theta_{n}}$ in the place of $\hat{\theta}$. – L.V.Rao Nov 6 '16 at 13:48

Some intuition behind the delta method:

The Delta method can be seen as combining two ideas:

1. An affine transformation of a multivariate normal random variable is multivariate normal and
2. Continuous, differentiable functions can be approximated locally by an affine transformation.

The 1st idea is from probability, the 2nd is from calculus. The loose intuition / argument goes:

• $\tilde{\boldsymbol{\theta}_n}$ is asymptotically normal (by assumption or by application of a central limit theorem in the case of a sample mean).
• In the small neighborhood where $\tilde{\boldsymbol{\theta}_n}$ has probability mass, $\mathbf{g}(\mathbf{x})$ looks like an affine transformation. (Note that function $\mathbf{g}$ has to satisfy certain conditions. E.g. this doesn't work at $x=0$ for $g(x) = x^2$.)
• $\tilde{\mathbf{y}} = \mathbf{g}(\tilde{\boldsymbol{\theta}})$ is an affine transformation of a multivariate normal random variable, hence $\tilde{\mathbf{y}}$ is also multivariate normal.

Idea 1: An affine transformation of a multivariate normal random variable is multivariate normal

Let's say we have $\tilde{\boldsymbol{\theta}}$ distributed multivariate normal with mean $\boldsymbol{\mu}$ and variance $V$. That is: $$\tilde{\boldsymbol{\theta}} \sim \mathcal{N}\left( \boldsymbol{\mu}, V\right)$$

Consider a linear transformation $A$ and consider the multivariate normal random variable defined by the linear transformation $A\tilde{\boldsymbol{\theta}}$. It's easy to show: $$A\tilde{\boldsymbol{\theta}} - A\boldsymbol{\mu} \sim \mathcal{N}\left(\mathbf{0}, AVA'\right)$$

Idea 2: Locally, any continuous, differentiable function looks affine.

An idea of calculus is if you zoom in enough on a continuous, differentiable function, it will look like a line. If we have some vector valued function $\mathbf{g}(\mathbf{x})$, in a small enough neighborhood around $\mathbf{c}$:

$$\mathbf{g}(\mathbf{c} + \boldsymbol{\epsilon}) \approx \mathbf{g}(\mathbf{c}) + \frac{\partial \mathbf{g}(\mathbf{c})}{\partial \mathbf{x}'} \;\boldsymbol{\epsilon}$$

Note that we have a problem doing this if any component of $\frac{\partial \mathbf{g}(\mathbf{c})}{\partial \mathbf{x}'}$ is zero. (eg. $g(x) = x^2$ at $x=0$.)

Putting it together:

If we know that $\tilde{\boldsymbol{\theta}} \sim \mathcal{N}\left( \boldsymbol{\mu}, V\right)$ and that function $\mathbf{g}(\mathbf{x})$ can be approximated around $\boldsymbol{\mu}$ by $\mathbf{g}(\boldsymbol{\mu}) + \frac{\partial \mathbf{g}(\boldsymbol{\mu})}{\partial \mathbf{x}'} \;\boldsymbol{\epsilon}$ then putting ideas (1) and (2) together:

$$\mathbf{g}\left( \tilde{\boldsymbol{\theta}} \right) - \mathbf{g}(\boldsymbol{\mu}) \sim \mathcal{N} \left( \mathbf{0}, \frac{\partial \mathbf{g}(\boldsymbol{\mu})}{\partial \mathbf{x}'} V \frac{\partial \mathbf{g}(\boldsymbol{\mu})}{\partial \mathbf{x}'} '\right)$$

This problem:

$$g(x) = x^2 \quad \quad g'(x) = 2 x$$

If $\sqrt{n}\left( \tilde{\theta} - \mu \right) \xrightarrow{d} \mathcal{N}(0, 1)$ Applying the delta method you get...

• I think, it is $\tilde{\theta}$ in the last equation – L.V.Rao Nov 6 '16 at 12:50
• $g(\tilde{\theta})-g(\mu)\sim \mathcal{N}(\cdot,\cdot)$, am I correct? – L.V.Rao Nov 6 '16 at 13:28
• @L.V.Rao Yeah. typos. – Matthew Gunn Nov 6 '16 at 13:37