# Taylor expansion for random variables

Let $$X_n$$ and $$Y_n$$ be random variables such that $$X_n-Y_n\overset{p}{\longrightarrow}0$$ as $$n\rightarrow\infty$$. Let $$f(.)$$ is a differentiable function. Is the following correct?

$$f(X_n) = f(Y_n) + f'(Y_n)(X_n-Y_n) + o_p(X_n-Y_n)$$ as $$n\rightarrow\infty$$

To gain some intuition about the issues, let's consider a sequence of bivariate random variables $$(X_n,Y_n)$$ that have simple distributions and a function $$f$$ that is easy to calculate with. We want to give your supposition a hard time in the sense of contemplating functions $$f$$ for which the Taylor approximation $$f(z+a) \approx f(z) + af^\prime(z)$$ is particularly poor. If it can withstand such examination then we will proceed to developing a proof; otherwise, we will have constructed a counterexample.

There are many ways to perform such an investigation. For instance, we could examine functions $$f$$ that, although differentiable, are not continuously differentiable. But perhaps the simplest is to examine functions whose derivative can become arbitrarily large, and then let the sequence $$(X_n,Y_n)$$ approach points where the derivative "blows up."

Consider, then, the function

$$f:(0,\infty)\to \mathbb{R};\ f(x) = \frac{1}{x},$$

chosen to be easy to compute with but to have a derivative

$$f^\prime(x) = -\frac{1}{x^2}$$

that diverges as $$x\to 0.$$

Let $$(X_n,Y_n)$$ be a sequence of random discrete random variables converging towards each other and towards $$0;$$ specifically, suppose

\eqalign{ \Pr\left((X_n,Y_n) = \left(\frac{2}{n}, \frac{1}{n}\right)\right) &= \frac{1}{2}\\ \Pr\left((X_n,Y_n) = \left(\frac{3}{n}, \frac{2}{n}\right)\right) &= \frac{1}{2}. }

Equivalently, $$nY_n-1$$ has a Bernoulli$$(1/2)$$ distribution and $$X_n = Y_n + 1/n.$$ Their difference converges to zero in any sense you care to use: In particular, since $$X_n-Y_n=1/n\to 0$$ everywhere, $$X_n-Y_n \overset{p}{\longrightarrow} 0.$$

Let's compute the difference $$f(X_n) - \left[ f(Y_n) + f^\prime(Y_n)(X_n-Y_n) \right].$$ This is a quantity you hope will be growing smaller in some sense: you want to be able to assert it is "$$o_p(X_n-Y_n).$$"

There are two possible values, each with probability $$1/2.$$ I will show the computations with a table, one row for each possibility:

$$\begin{array}{cc|cccc} X_n & Y_n & f(X_n) & f(Y_n) & f^\prime(Y_n) & f(X_n) - \left[ f(Y_n) + f^\prime(Y_n)(X_n-Y_n) \right] \\ \hline \frac{2}{n} & \frac{1}{n} & \frac{n}{2} & n & -n^2 & \frac{n}{2} - [n - n^2\frac{1}{n}] = \frac{n}{2} \\ \frac{3}{n} & \frac{2}{n} & \frac{n}{3} & \frac{n}{2} & -\frac{n^2}{4} & \frac{n}{3} - [\frac{n}{2} - \frac{n^2}{4}\frac{1}{n}] = \frac{n}{12} \end{array}$$

Because the values $$n/2$$ and $$n/12$$ diverge everywhere, there is no possible definition of "$$o_p$$" that could make the statement true.

The insight provided by this simple example, and variations on it (in which $$X_n$$ and $$Y_n$$ may be independent or $$f$$ may be defined everywhere, for instance) is you cannot allow $$X_n$$ and $$Y_n$$ to have any appreciable probability of attaining values where the Taylor approximation becomes arbitrarily poor.