3
$\begingroup$

In this post, the accepted answer states that we need certain conditions before a second order Taylor series approximation is robust, due to the fact that the variance does not control higher moments.

The author goes on to say that

Therefore, in general, the Taylor series method fails even for 3rd degree polynomials. Iterating this argument shows you cannot expect the Taylor series method to provide accurate results, even for polynomials, unless all moments of your random variable are well controlled.

I'm not fully convinced of this statement and I'm wondering do we really need all moments to be well-controlled for a valid second order Taylor expansion. Because if we look at the formula for the remainder of a Taylor series truncated at second order, such here or here, it only uses the third moment $$ R_2(X) = \frac{f^{(3)}(\zeta)}{3!}E[(X-\mu)^3], $$ and the derivation of this formula does not use the higher moments of the random variable.

So have I overlooked something or is it indeed sufficient to have control of the third moment to ensure the Taylor series approximation is valid? Why would we need control of all moments as the accepted answer in the other post states when the remainder is derived without using higher order moments?

Edit: The formula for the 2nd order Taylor expansion is $$ E[f(X)] = f(\mu) + \frac{1}{2}f''(\mu)E[(X-\mu)^2] + R_2(X). $$

As far as I can see the higher order moments ($E[(X-\mu)^k]$ with $k \ge 4$) have not been used to derive the remainder in that formula so I believe that the formula is accurate as long as only the third moment is well-controlled, e.g. $E[(X-\mu)^3] \ll E[(X-\mu)^2]$.

So am I incorrect or is the linked answer, where the author says all moments of the random variable need to be well controlled, incorrect? If I am incorrect please demonstrate why the higher order moments need to be well-controlled for the above formula to be accurate.

$\endgroup$
13
  • $\begingroup$ Taylor expansion is guaranteed to works because the order of the error bounds from above the error of higher order terms. If we have undefined or diverging moments (which we can have) higher than the third one, that this is not guaranteed anymore. $\endgroup$
    – Three Diag
    Mar 1, 2021 at 17:34
  • 1
    $\begingroup$ The answer depends equally on the terms of the Taylor expansion and on the moments of $X.$ Perhaps it would be more fruitful to ask "how accurate is the Taylor expansion" [for whatever purpose you are using it for] rather than to ask about "validity," because the latter is not a well-defined or even quantitative criterion. $\endgroup$
    – whuber
    Mar 1, 2021 at 17:48
  • $\begingroup$ @ThreeDiag My point is that if we consider $E[f(X)] = f(\mu) + \frac{1}{2} f''(\mu) E[(X-\mu)^2] + R_2(X)$, the formula for $R_2(X)$ is derived without using higher moments of $X$, therefore, even if the higher moments are diverging, it won't make a difference: $E[f(X)] = f(\mu) + \frac{1}{2} f''(\mu) E[(X-\mu)^2]$ will still be an accurate approximation as long as it holds that the third moment (which appears in $R_2(X)$) is sufficiently smaller than the second moment. $\endgroup$
    – sonicboom
    Mar 2, 2021 at 10:03
  • $\begingroup$ @whuber See my comment above regarding my condition on the moments of $X$ - essentially, I am saying that if we have/assume the third moment of $X$ is sufficiently smaller than the second moment, then the truncated second order Taylor expansion is a good approximation (ignoring $f(\mu)$ and $f''(\mu)$ for the sake of argument). This is because the formula $E[f(X)] = f(\mu) + \frac{1}{2} f''(\mu) E[(X-\mu)^2] + R_2(X)$ does not make us of moments higher than the third moment. $\endgroup$
    – sonicboom
    Mar 2, 2021 at 10:08
  • 1
    $\begingroup$ When there's appreciable chance that $|X-\mu|\gg 1,$ that assumption (which requires some absolute values, btw) will be violated. $\endgroup$
    – whuber
    Mar 10, 2021 at 15:16

1 Answer 1

1
$\begingroup$

The short answer is that in general, yes, we do need to control all moments (and ensure the function $f$ is sufficiently "nice"). The short reason that the logic above fails is that the location $\zeta$ is a function of $x$.

Take a close look at the definition of the explicit remainder terms in Taylor's theorem. It guarantees only that $\zeta$ lies between $\mu$ and $x$, and so the formula for the expectation of the remainder should be

$$ \mathbb{E}[R_2(X)] = \mathbb{E}\left[ \frac{f^{(3)}(\zeta(X))}{3!}(X-\mu)^3\right]. $$

The term inside the expectation remains a third-degree polynomial in $X$ if and only if $f$ itself is a third-degree polynomial. (Indeed, the remainder term is, by definition, the difference between the function and it's Taylor approximation.)

A simple counterexample

Consider the random variable $X$ with pdf

$$ p_X(x) = c \cdot \frac{1}{1+ x^6} $$

where $c$ is the normalization constant. It's an exercise to check that $X$ has finite first through fourth moments. Now, taking $f(x) = x^3$, we find

$$ \mathrm{Var\left[f\bigl(\frac{1}{n}X\bigr)\right]}=\mathbb{E}\left[f\bigl(\frac{1}{n}X\bigr)^2\right] = \frac{c}{n^3} \int_{-\infty}^\infty \frac{x^6}{1+x^6}\mathrm{d}x = + \infty $$

for any $n\in \mathbb{N}$, while the second-order Taylor approximation is uniformly zero.

$\endgroup$
3
  • $\begingroup$ Just to add, we can obtain a valid 2nd order Taylor approximation of $f(X)$ as long as $|f^3(x)| < C$ for some constant $C > 0$ for all $x$. $\endgroup$
    – sonicboom
    Jul 14, 2021 at 8:43
  • $\begingroup$ I think you mean $|f^{(3)}(x)| < C$ (that is, bounded third derivative). But that alone is not strong enough, since you'll also need $\mathbb{E}[(X-\mu)^3] \ll 1$. Anyway, yes, there are conditions under which the Taylor series approximation is valid that avoid control of any moments—trivially, the Taylor series is valid if $f(x) = 0$. But in general (and most applications), you need to control of both the moments of $X$ and the regularity of $f$. Even a slow-growth function like $log(1+X)$ for $X>0$ won't have a valid Taylor series approximation without strong control on moments. $\endgroup$
    – Mike McCoy
    Jul 14, 2021 at 15:27
  • $\begingroup$ I do think that I've answered the question. :) The key point is that we cannot pull the $f'''$ outside of the expectation in the remainder term because it depends on $X$. This is where the higher-order moments of $X$ appear in the remainder. $\endgroup$
    – Mike McCoy
    Jul 14, 2021 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.