Taking the expectation of Taylor series (especially the remainder)

Question

My question concerns trying to justify a widely-used method, namely taking the expected value of Taylor Series. Assume we have a random variable $X$ with positive mean $\mu$ and variance $\sigma^2$. Additionally, we have a function, say, $\log(x)$.

Doing Taylor Expansion of $\log X$ around the mean, we get $$ \log X = \log\mu + \frac{X - \mu}{\mu} - \frac12 \frac{(X-\mu)^2}{\mu^2} + \frac13 \frac{(X - \mu)^3}{\xi_X^3}, $$ where, as usual, $\xi_X$ is s.t. $|\xi_X - \mu| < |X - \mu|$.

If we take an expectation, we will get an approximate equation which people usually refer to as something self-apparent (see the $\approx$ sign in the first equation here): $$ \mathbb{E}\log X \approx \log \mu - \frac12 \frac{\sigma^2}{\mu^2} $$

QUESTION: I'm interested in how to prove that the expected value of the remainder term is actually negligible, i.e. $$ \mathbb{E}\left[\frac{(X - \mu)^3}{\xi_X^3}\right] = o(\sigma^2) $$ (or, in other words, $\mathbb{E}\bigl[o(X-\mu)^2\bigr] = o\bigl(\mathbb{E}\bigl[(X-\mu)^2\bigr]\bigr)$).

What I tried to do: assuming that $\sigma^2 \to 0$ (which, in turn, means $X \to \mu$ in $\mathbb{P}$), I tried to split the integral into two, surrounding $\mu$ with some $\varepsilon$-vicinity $N_\varepsilon$: $$ \int_\mathbb{R} p(x)\frac{(x-\mu)^3}{\xi_x^3} \,dx = \int_{x \in N_\varepsilon} \ldots dx + \int_{x \notin N_\varepsilon} \ldots dx $$

The first one can be bounded due to the fact that $0 \notin N_\varepsilon$ and thus $1/\xi^3$ doesn't bother. But with the second one we have two concurring facts: on the one hand $$ \mathbb{P}(|X - \mu| > \varepsilon) \to 0 $$ (as $\sigma^2 \to 0$). But on the other hand, we don't know what to do with $1/\xi^3$.

Another possibility could be to try using the Fatou's lemma, but I can't figure out how.

Will appreciate any help or hint. I realize that this is sort of a very technical question, but I need to go through it in order to trust this "Taylor-expectation" method. Thanks!

P.S. I checked out here, but seems it's a bit of another stuff.

Answer 1

You are right to be skeptical of this approach. The Taylor series method does not work in general, although the heuristic contains a kernel of truth. To summarize the technical discussion below,

• Strong concentration implies that the Taylor series method works for nice functions
• Things can and will go dramatically wrong for heavy-tailed distributions or not-so-nice functions

As Alecos's answer indicates, this suggests that the Taylor-series method should be scrapped if your data might have heavy tails. (Finance professionals, I'm looking at you.)

As Elvis noted, key problem is that the variance does not control higher moments. To see why, let's simplify your question as much as possible to get to the main idea.

Suppose we have a sequence of random variables $X_n$ with $\sigma(X_n)\to 0$ as $n\to \infty$.

Q: Can we guarantee that $\mathbb{E}[|X_n-\mu|^3] = o(\sigma^2(X_n))$ as $n\to \infty?$

Since there are random variables with finite second moments and infinite third moments, the answer is emphatically no. Therefore, in general, the Taylor series method fails even for 3rd degree polynomials. Iterating this argument shows you cannot expect the Taylor series method to provide accurate results, even for polynomials, unless all moments of your random variable are well controlled.

What, then, are we to do? Certainly the method works for bounded random variables whose support converges to a point, but this class is far too small to be interesting. Suppose instead that the sequence $X_n$ comes from some highly concentrated family that satisfies (say)

$$\mathbb{P}\left\{ |X_n-\mu|> t\right\} \le \mathrm{e}^{- C n t^2} \tag{1}$$

for every $t>0$ and some $C>0$. Such random variables are surprisingly common. For example when $X_n$ is the empirical mean

$$ X_n := \frac{1}{n} \sum_{i=1}^n Y_i$$

of nice random variables $Y_i$ (e.g., iid and bounded), various concentration inequalities imply that $X_n$ satisfies (1). A standard argument (see p. 10 here) bounds the $p$th moments for such random variables:

$$ \mathbb{E}[|X_n-\mu|^p] \le \left(\frac{p}{2 C n}\right)^{p/2}.$$

Therefore, for any "sufficiently nice" analytic function $f$ (see below), we can bound the error $\mathcal{E}_m$ on the $m$-term Taylor series approximation using the triangle inequality

$$ \mathcal{E}_m:=\left|\mathbb{E}[f(X_n)] - \sum_{p=0}^m \frac{f^{(p)}(\mu)}{p!} \mathbb{E}(X_n-\mu)^p\right|\le \tfrac{1}{(2 C n)^{(m+1)/2}} \sum_{p=m+1}^\infty |f^{(p)}(\mu)| \frac{p^{p/2}}{p!}$$

when $n>C/2$. Since Stirling's approximation gives $p! \approx p^{p-1/2}$, the error of the truncated Taylor series satisfies

$$ \mathcal{E}_m = O(n^{-(m+1)/2}) \text{ as } n\to \infty\quad \text{whenever} \quad \sum_{p=0}^\infty p^{(1-p)/2 }|f^{(p)}(\mu)| < \infty \tag{2}.$$

Hence, when $X_n$ is strongly concentrated and $f$ is sufficiently nice, the Taylor series approximation is indeed accurate. The inequality appearing in (2) implies that $f^{(p)}(\mu)/p! = O(p^{-p/2})$, so that in particular our condition requires that $f$ is entire. This makes sense because (1) does not impose any boundedness assumptions on $X_n$.

Let's see what can go wrong when $f$ is has a singularity (following whuber's comment). Suppose that we choose $f(x)=1/x$. If we take $X_n$ from the $\mathrm{Normal}(1,1/n)$ distribution truncated between zero and two, then $X_n$ is sufficiently concentrated but $\mathbb{E}[f(X_n)] = \infty$ for every $n$. In other words, we have a highly concentrated, bounded random variable, and still the Taylor series method fails when the function has just one singularity.

A few words on rigor. I find it nicer to present the condition appearing in (2) as derived rather than a deus ex machina that's required in a rigorous theorem/proof format. In order to make the argument completely rigorous, first note that the right-hand side in (2) implies that

$$\mathbb{E}[|f(X_n)|] \le \sum_{i=0}^\infty \frac{|f^{(p)}(\mu)|}{p!} \mathbb{E}[|X_n-\mu|^p]< \infty$$

by the growth rate of subgaussian moments from above. Thus, Fubini's theorem provides

$$ \mathbb{E}[f(X_n)] = \sum_{i=0}^\infty \frac{f^{(p)}(\mu)}{p!} \mathbb{E}[(X_n-\mu)^p]$$

The rest of the proof proceeds as above.

Answer 2

Although my answer will nowhere approach the level of mathematical sophistication of the other answers, I decided to post it because I believe it has something to contribute -although the result will be "negative", as they say.

In a light tone, I would say that the OP is "risk-averse", (as most people are, as well as science itself), because the OP requires a sufficient condition for the 2nd-order Taylor series expansion approximation to "be acceptable". But it is not a necessary condition.

Firstly, a necessary but not sufficient pre-requisite for the expected value of the Remainder to be of lower order than the variance of the r.v., as the OP requires, is that the series converges in the first place. Should we just assume convergence? No.

The general expression we examine is

$$ E\Big[g(Y)\Big] = \int_{-\infty}^{\infty}f_Y(y)\Big[\sum_{i=0}^{\infty}g^{(i)}(\mu)\frac{(y-\mu)^i}{i!}\Big]dy \qquad [1]$$

As Loistl (1976) states, referencing Gemignani's "Calculus and Statistics" book (1978, p. 170), a condition for convergence of the infinite sum is (an application of the ratio test for convergence)

$$y-\mu < |y-\mu|<\lim_{i\rightarrow \infty}\left | \left(\frac {g^{(i)}(\mu)}{g^{(i+1)}(\mu)}(i+1)\right)\right| \qquad [2]$$

...where $\mu$ is the mean of the r.v. Although this too is a sufficient condition (the ratio test is inconclusive if the above relation holds with equality), the series will diverge if the inequality holds in the other direction.

Loistl examined three specific functional forms for $g()$, the exponential, the power, and the logarithm (his paper is in the field of Expected Utility and Portfolio Choice, so he tested the standard functional forms used to represent a concave utility function). For these functional forms, he found that only for the exponential functional form no restrictions on $y-\mu$ were imposed. On the contrary, for the power, and for the logarithmic case (where we already have $0 <y$), we find that the validity of inequality $[2]$ is equivalent to $$y-\mu < \mu \Rightarrow 0 < y < 2\mu$$

This means that if our variable varies outside this range, the Taylor expansion having as expansion center the variable's mean will diverge.

So: for some functional forms, the value of a function at some point of its domain equals its infinite Taylor expansion, no matter how far this point is from the expansion center. For other functional forms (logarithm included), the point of interest should lie somewhat "close" to the chosen center of expansion. In the case where we have a r.v., this translates to a restriction on the theoretical support of the variable (or an examination of its empirically observed range).

Loitl, using numerical examples, showed also that increasing the order of the expansion before truncation could make matters worse for the accuracy of the approximation. We must note that empirically, time-series of observed variables in the financial sector do exhibit variability larger than the one required by the inequality. So Loitl went on to advocate that the Taylor series approximation methodology should be scrapped entirely, regarding Portfolio Choice Theory.

The rebound came 18 years later from Hlawitschka (1994). The valuable insight and result here was, and I quote

...although a series may ultimately converge, little can be said about any of its partial series; convergence of a series does not imply that the terms immediately decrease in size or that any particular term is sufficiently small to be ignored. Indeed, it is possible, as demonstrated here, that a series may appear to diverge before ultimately converging in the limit. The quality of moment approximations to expected utility that are based upon the first few terms of a Taylor series, therefore, cannot be determined by the convergence properties of the infinite series. This is an empirical issue, and empirically, two-moment approximations to the utility functions studied here perform well for the task of portfolio selection. Hlawitschka (1994)

By example, Hlawitschka showed that the 2nd-order approximation was "successful" whether the Taylor series converged or not, but he also verified Lotl's result, that increasing the order of the approximation may make it worse. But there is a qualifier for this success: In Portfolio Choice, Expected Utility is used to rank securities and other financial products. It is an ordinal measure, not cardinal. So what Hlawitschka found is that the 2nd-order approximation preserved the ranking of different securities, compared to the ranking stemming from the exact value of $E(g(Y)$, and not that it always gave quantitative results that where sufficiently close to this exact value (see his table A1 in p. 718).

So where does that leave us? In limbo, I'd say. It appears that both in theory and in empirics, the acceptability of the 2nd-order Taylor approximation depends critically on many different aspects of the specific phenomenon under study and the scientific methodology employed -it depends on the theoretical assumptions, on the functional forms used, on the observed variability of the series...

But let's end this positively: nowadays, computer power substitutes for a lot of things. So we could simulate and test the validity of the 2nd-order approximation, for a wide range of values of the variable cheaply, whether we work on a theoretical, or an empirical problem.

Answer 3

Not an actual answer, but an example to show that things are not so nice, and that extra hypotheses are needed to make this result true.

Define $X_n$ as a mixture between a uniform $U\left( \left[ -{1\over n} ; {1\over n} \right] \right)$ and a normal $\mathcal N({n \over n-1}, {1\over n})$, the uniform component being chosen with probability $1\over n$, and the normal with probability $1 -{1\over n} = {n-1 \over n}$. You have $E(X_n) = 1$ and its variance converges to $0$ when $n$ goes to infinity, as $$E\left(X_n^2\right) = {1\over 3 n^2} \times {1\over n} + \left(\left({n \over n-1}\right)^2+{1\over n}\right)\times{n-1 \over n},$$ if I am not mistaking.

Now define $f(x) = 1/x$ (and $f(0) = 0$ or whatever). The random variables $f(X_n)$ are well defined but does not have an expected value, as $$ \int_{-{1\over n}}^{1\over n} {1\over x} \mathrm dx$$ is not defined, no matter how big $n$ is.

My conclusion is that you clearly need hypotheses on either the global behavior of $f$ or – more likely, more elegantly – on the speed at which the density of $X_n$ decays when you’re far from the expected value. I am sure that such hypotheses can be found in the classic literature (and even in textbooks), unfortunately my training was not in statistics and I still struggle with the literature myself... anyway I hope this helped.

PS. Isn’t this example a counter-example to Nick’s answer? Who’s wrong then?

Answer 4

This is not a complete answer, just a different way of arriving at the second order approximation.

I think the best way to go is to use Cauchy's mean value theorem, rather than work with the remainder term of a Taylor series. If we apply it one time we have $$f(X)=f(\mu)+f'(\xi_1)(X-\mu)$$

for some $X\leq\xi_1 \leq \mu$ when $X \leq \mu$ or $X\geq\xi_1 \geq \mu$ when $X \geq \mu$. We now apply the mean value theorem again to $ f'(\xi_1)$ and we have

$$ f'(\xi_1)= f'(\mu) + f''(\xi_2)( \xi_1-\mu)$$

for some $X\leq\xi_1\leq\xi_2\leq\mu$ when $X\leq\mu$ or $X\geq\xi_1\geq \xi_2 \geq\mu$ when $X\geq\mu$. putting this into the first fomula gives

$$f(X)=f(\mu)+ f'(\mu) (X-\mu) + f''(\xi_2)( \xi_1-\mu) (X-\mu)$$

Note that this result only requires that $f$ is continuous and twice differentiable between $X$ and $\mu$. However this applies only for a fixed $X$, and changing $X$ will mean a corresponding change in $\xi_i$. The second order delta method can be seen as making the global assumption that $\xi_1-\mu=\frac{1}{2}(X-\mu)$ and $\xi_2=\mu$ over the entire range of the support of $X$, or at least over the region of high probability mass.