29
$\begingroup$

Is it possible to test for finiteness (or existence) of the variance of a random variable given a sample? As a null, either {the variance exists and is finite} or {the variance does not exist/is infinite} would be acceptable. Philosophically (and computationally), this seems very strange because there should be no difference between a population without finite variance, and one with a very very large variance (say > $10^{400}$), so I am not hopeful this problem can be solved.

One approach that had been suggested to me was via the Central Limit Theorem: assuming the samples are i.i.d., and the population has finite mean, one could check, somehow, whether the sample mean has the right standard error with increasing sample size. I'm not sure I believe this method would work, though. (In particular, I don't see how to make it into a proper test.)

$\endgroup$
13
$\begingroup$

No, this is not possible, because a finite sample of size $n$ cannot reliably distinguish between, say, a normal population and a normal population contaminated by a $1/N$ amount of a Cauchy distribution where $N$ >> $n$. (Of course the former has finite variance and the latter has infinite variance.) Thus any fully nonparametric test will have arbitrarily low power against such alternatives.

$\endgroup$
  • 3
    $\begingroup$ this is a very good point. however, don't most hypothesis tests have arbitrarily low power against some alternative? e.g. a test for zero mean will have very low power when given a sample from a population with mean $\epsilon$ for $0 \lt |\epsilon|$ small. I am still left wondering whether such a test can be sanely constructed at all, much less whether it has low power in some cases. $\endgroup$ – shabbychef Sep 9 '10 at 16:11
  • 2
    $\begingroup$ also, 'polluted' distributions like the one you cite always seemed to me to be at odds with the idea of being 'identically distributed'. Perhaps you would agree. It seems that saying samples are drawn i.i.d. from some distribution without stating the distribution is meaningless (well, the 'independently' part of i.i.d. is meaningful). $\endgroup$ – shabbychef Sep 9 '10 at 16:14
  • 2
    $\begingroup$ (1) You're right about the low power, but the problem here (it seems to me) is that there is no gradual step from "finite" to "infinite": the problem seems not to have a natural scale to tell us what constitutes a "small" departure from the null compared to a "large" departure. (2) The distributional form is independent of considerations of iid. I don't mean that, say, 1% of the data will come from a Cauchy and 99% from a Normal. I mean that 100% of the data come from a distribution that is almost normal but has Cauchy tails. In this sense the data can be iid for a contaminated distribution. $\endgroup$ – whuber Sep 9 '10 at 17:39
  • 2
    $\begingroup$ Has anybody read this paper? sciencedirect.com/science/article/pii/S0304407615002596 $\endgroup$ – Christoph Hanck Jan 23 '16 at 9:04
  • 2
    $\begingroup$ @shabbychef if every observation arises from the exact same mixture process they're identically distributed, each as a draw from the corresponding mixture distribution. If some observations are necessarily from one process and others are necessarily from a different process (observations 1 to 990 are normal and observations 991 to 1000 are Cauchy, say), then they're not identically distributed (even though the combined sample may be indistinguishable from a 99%-1% mixture). This essentially comes down to the model of the process you're using. $\endgroup$ – Glen_b Jun 30 '17 at 1:28
16
$\begingroup$

You cannot be certain without knowing the distribution. But there are certain things you can do, such as looking at what might be called the "partial variance", i.e. if you have a sample of size $N$, you draw the variance estimated from the first $n$ terms, with $n$ running from 2 to $N$.

With a finite population variance, you hope that the partial variance soon settles down close to the population variance.

With an infinite population variance, you see jumps up in the partial variance followed by slow declines until the next very large value appears in the sample.

This is an illustration with Normal and Cauchy random variables (and a log scale) Partial Variance

This may not help if the shape of your distribution is such that a much larger sample size than you have is needed to identify it with sufficient confidence, i.e. where very large values are fairly (but not extremely) rare for a distribution with finite variance, or are extremely rare for a distribution with infinite variance. For a given distribution there will be sample sizes which are more likely than not to reveal its nature; conversely, for a given sample size, there are distributions which are more likely than not to disguise their natures for that size of sample.

$\endgroup$
  • 4
    $\begingroup$ +1 I like this because (a) a graphic usually reveals much more than a test and (b) it is practical. I'm a little concerned that it has an arbitrary aspect: its appearance will depend (strongly, perhaps) on the order in which the data are given. When the "partial variance" is due to one or two extreme values, and they come near the beginning, this graphic might be deceptive. I wonder whether there's a good solution to this problem. $\endgroup$ – whuber Nov 2 '11 at 17:38
  • 1
    $\begingroup$ +1 for great graphic. Really solidifies the concept of "no variance" in the Cauchy distribution. @whuber: Sorting the data in all possible permutations, running the test for each, and taking some kind of average? Not very computationally efficient, I'll grant you :) but maybe you could just chose a handful of random permutations? $\endgroup$ – naught101 Jul 27 '12 at 5:03
  • 2
    $\begingroup$ @naught101 Averaging over all permutations won't tell you anything, because you will get a perfectly horizontal line. Perhaps I misunderstand what you mean? $\endgroup$ – whuber Jul 27 '12 at 13:34
  • 1
    $\begingroup$ @whuber: I actually meant taking the average of some kind of test for convergence, not the graph itself. But I'll grant it's a pretty vague idea, and that's largely because I have no idea what I'm talking about :) $\endgroup$ – naught101 Jul 28 '12 at 1:42
7
$\begingroup$

Here's another answer. Suppose you could parametrize the problem, something like this:

$$ H_{0}:\ X \sim t(\mathtt{df}=3)\mathrm{\ versus\ } H_{1}:\ X \sim t(\mathtt{df}=1). $$

Then you could do an ordinary Neyman-Pearson likelihood ratio test of $H_{0}$ versus $H_{1}$. Note that $H_{1}$ is Cauchy (infinite variance) and $H_{0}$ is the usual Student's $t$ with 3 degrees of freedom (finite variance) which has PDF: $$ f(x|\nu) = \frac{\Gamma\left(\frac{\nu + 1}{2}\right)}{\sqrt{\nu\pi}\Gamma\left(\frac{\nu}{2}\right)}\left(1 + \frac{x^{2}}{\nu} \right)^{-\frac{\nu + 1}{2}}, $$

for $-\infty < x < \infty$. Given simple random sample data $x_{1},x_{2},\ldots,x_{n}$, the likelihood ratio test rejects $H_{0}$ when $$ \Lambda(\mathbf{x}) = \frac{\prod_{i=1}^{n}f(x_{i}|\nu = 1)}{\prod_{i=1}^{n}f(x_{i}|\nu = 3)} > k, $$ where $k \geq 0$ is chosen such that $$ P(\Lambda(\mathbf{X}) > k\,|\nu = 3) = \alpha. $$

It's a little bit of algebra to simplify $$ \Lambda(\mathbf{x}) = \left(\frac{\sqrt{3}}{2}\right)^{n}\prod_{i = 1}^{n}\frac{\left(1 + x_{i}^{2}/3 \right)^{2}}{1 + x_{i}^{2}}. $$

So, again, we get a simple random sample, calculate $\Lambda(\mathbf{x})$, and reject $H_{0}$ if $\Lambda(\mathbf{x})$ is too big. How big? That's the fun part! It's going to be hard (impossible?) to get a closed form for the critical value, but we could approximate it as close as we like, for sure. Here's one way to do it, with R. Suppose $\alpha = 0.05$, and for laughs, let's say $n = 13$.

We generate a bunch of samples under $H_{0}$, calculate $\Lambda$ for each sample, and then find the 95th quantile.

set.seed(1)
x <- matrix(rt(1000000*13, df = 3), ncol = 13)
y <- apply(x, 1, function(z) prod((1 + z^2/3)^2)/prod(1 + z^2))
quantile(y, probs = 0.95)

This turns out to be (after some seconds) on my machine to be $\approx 12.8842$, which after multiplied by $(\sqrt{3}/2)^{13}$ is $k \approx 1.9859$. Surely there are other, better, ways to approximate this, but we're just playing around.

In summary, when the problem is parametrizable you can set up a hypothesis test just like you would in other problems, and it's pretty straightforward, except in this case for some tap dancing near the end. Note that we know from our theory the test above is a most powerful test of $H_{0}$ versus $H_{1}$ (at level $\alpha$), so it doesn't get any better than this (as measured by power).

Disclaimers: this is a toy example. I do not have any real-world situation in which I was curious to know whether my data came from Cauchy as opposed to Student's t with 3 df. And the original question didn't say anything about parametrized problems, it seemed to be looking for more of a nonparametric approach, which I think was addressed well by the others. The purpose of this answer is for future readers who stumble across the title of the question and are looking for the classical dusty textbook approach.

P.S. it might be fun to play a little more with the test for testing $H_{1}:\nu \leq 1$, or something else, but I haven't done that. My guess is that it'd get pretty ugly pretty fast. I also thought about testing different types of stable distributions, but again, it was just a thought.

$\endgroup$
  • 2
    $\begingroup$ estimating the $\alpha$ in stable distributions is notoriously difficult. $\endgroup$ – shabbychef Nov 2 '11 at 16:49
  • 1
    $\begingroup$ You could test also that $H_1:\nu\leq 2$, because T-dist has finite variance only for $\nu>2$. $\endgroup$ – probabilityislogic Nov 2 '11 at 23:39
  • 2
    $\begingroup$ Re: $\alpha$, I didn't know it was notoriously difficult, but it sounds right, thanks. @probability, you are right, and the only reason I picked 3 versus 1 was because it meant less fractions. And BTW, I liked probability's answer better than mine (+1). $\endgroup$ – user1108 Nov 3 '11 at 0:24
  • 1
    $\begingroup$ maybe I misremembered the result: something about tail index estimation when $\alpha$ is near 2; the paper is by Weron, I think. That aside, testing $\alpha = 2$ against a sum-stable alternative is a kind of normality test! Such tests usually reject given sufficient (real) data: see e.g. stats.stackexchange.com/questions/2492/… $\endgroup$ – shabbychef Nov 3 '11 at 17:04
6
$\begingroup$

In order to test such a vague hypothesis, you need to average out over all densities with finite variance, and all densities with infinite variance. This is likely to be impossible, you basically need to be more specific. One more specific version of this and have two hypothesis for a sample $D\equiv Y_{1},Y_{2},\dots,Y_{N}$:

  1. $H_{0}:Y_{i}\sim Normal(\mu,\sigma)$
  2. $H_{A}:Y_{i}\sim Cauchy(\nu,\tau)$

One hypothesis has finite variance, one has infinite variance. Just calculate the odds:

$$\frac{P(H_{0}|D,I)}{P(H_{A}|D,I)}=\frac{P(H_{0}|I)}{P(H_{A}|I)}\frac{\int P(D,\mu,\sigma|H_{0},I)d\mu d\sigma}{\int P(D,\nu,\tau|H_{A},I)d\nu d\tau} $$

Where $\frac{P(H_{0}|I)}{P(H_{A}|I)}$ is the prior odds (usually 1)

$$P(D,\mu,\sigma|H_{0},I)=P(\mu,\sigma|H_{0},I)P(D|\mu,\sigma,H_{0},I)$$ And $$P(D,\nu,\tau|H_{A},I)=P(\nu,\tau|H_{A},I)P(D|\nu,\tau,H_{A},I)$$

Now you normally wouldn't be able to use improper priors here, but because both densities are of the "location-scale" type, if you specify the standard non-informative prior with the same range $L_{1}<\mu,\tau<U_{1}$ and $L_{2}<\sigma,\tau<U_{2}$, then we get for the numerator integral:

$$\frac{\left(2\pi\right)^{-\frac{N}{2}}}{(U_1-L_1)log\left(\frac{U_2}{L_2}\right)}\int_{L_2}^{U_2}\sigma^{-(N+1)}\int_{L_1}^{U_1} exp\left(-\frac{N\left[s^{2}-(\overline{Y}-\mu)^2\right]}{2\sigma^{2}}\right)d\mu d\sigma$$

Where $s^2=N^{-1}\sum_{i=1}^{N}(Y_i-\overline{Y})^2$ and $\overline{Y}=N^{-1}\sum_{i=1}^{N}Y_i$. And for the denominator integral:

$$\frac{\pi^{-N}}{(U_1-L_1)log\left(\frac{U_2}{L_2}\right)}\int_{L_2}^{U_2}\tau^{-(N+1)}\int_{L_1}^{U_1} \prod_{i=1}^{N}\left(1+\left[\frac{Y_{i}-\nu}{\tau}\right]^{2}\right)^{-1}d\nu d\tau$$

And now taking the ratio we find that the important parts of the normalising constants cancel and we get:

$$\frac{P(D|H_{0},I)}{P(D|H_{A},I)}=\left(\frac{\pi}{2}\right)^{\frac{N}{2}}\frac{\int_{L_2}^{U_2}\sigma^{-(N+1)}\int_{L_1}^{U_1} exp\left(-\frac{N\left[s^{2}-(\overline{Y}-\mu)^2\right]}{2\sigma^{2}}\right)d\mu d\sigma}{\int_{L_2}^{U_2}\tau^{-(N+1)}\int_{L_1}^{U_1} \prod_{i=1}^{N}\left(1+\left[\frac{Y_{i}-\nu}{\tau}\right]^{2}\right)^{-1}d\nu d\tau}$$

And all integrals are still proper in the limit so we can get:

$$\frac{P(D|H_{0},I)}{P(D|H_{A},I)}=\left(\frac{2}{\pi}\right)^{-\frac{N}{2}}\frac{\int_{0}^{\infty}\sigma^{-(N+1)}\int_{-\infty}^{\infty} exp\left(-\frac{N\left[s^{2}-(\overline{Y}-\mu)^2\right]}{2\sigma^{2}}\right)d\mu d\sigma}{\int_{0}^{\infty}\tau^{-(N+1)}\int_{-\infty}^{\infty} \prod_{i=1}^{N}\left(1+\left[\frac{Y_{i}-\nu}{\tau}\right]^{2}\right)^{-1}d\nu d\tau}$$

The denominator integral cannot be analytically computed, but the numerator can, and we get for the numerator: $$\int_{0}^{\infty}\sigma^{-(N+1)}\int_{-\infty}^{\infty} exp\left(-\frac{N\left[s^{2}-(\overline{Y}-\mu)^2\right]}{2\sigma^{2}}\right)d\mu d\sigma=\sqrt{2N\pi}\int_{0}^{\infty}\sigma^{-N} exp\left(-\frac{Ns^{2}}{2\sigma^{2}}\right)d\sigma$$

Now make change of variables $\lambda=\sigma^{-2}\implies d\sigma = -\frac{1}{2}\lambda^{-\frac{3}{2}}d\lambda$ and you get a gamma integral:

$$-\sqrt{2N\pi}\int_{\infty}^{0}\lambda^{\frac{N-1}{2}-1} exp\left(-\lambda\frac{Ns^{2}}{2}\right)d\lambda=\sqrt{2N\pi}\left(\frac{2}{Ns^{2}}\right)^{\frac{N-1}{2}}\Gamma\left(\frac{N-1}{2}\right)$$

And we get as a final analytic form for the odds for numerical work:

$$\frac{P(H_{0}|D,I)}{P(H_{A}|D,I)}=\frac{P(H_{0}|I)}{P(H_{A}|I)}\times\frac{\pi^{\frac{N+1}{2}}N^{-\frac{N}{2}}s^{-(N-1)}\Gamma\left(\frac{N-1}{2}\right)}{\int_{0}^{\infty}\tau^{-(N+1)}\int_{-\infty}^{\infty} \prod_{i=1}^{N}\left(1+\left[\frac{Y_{i}-\nu}{\tau}\right]^{2}\right)^{-1}d\nu d\tau}$$

So this can be thought of as a specific test of finite versus infinite variance. We could also do a T distribution into this framework to get another test (test the hypothesis that the degrees of freedom is greater than 2).

$\endgroup$
  • 1
    $\begingroup$ When you started to integrate, you introduced a term $s^2$. It persists through the final answer. What is it? $\endgroup$ – whuber Apr 3 '11 at 22:13
  • 2
    $\begingroup$ @whuber - $s$ is the standard deviation MLE, $s^2=N^{-1}\sum_{i=1}^{N}(Y_i-\overline{Y})^2$. I thought it was the usual notation for standard deviation, just as $\overline{Y}$ is usual for average - which I have incorrectly written as $\overline{x}$, will edit accordingly $\endgroup$ – probabilityislogic Apr 3 '11 at 22:39
5
$\begingroup$

The counterexample is not relevant to the question asked. You want to test the null hypothesis that a sample of i.i.d. random variables is drawn from a distribution having finite variance, at a given significance level. I recommend a good reference text like "Statistical Inference" by Casella to understand the use and the limit of hypothesis testing. Regarding h.t. on finite variance, I don't have a reference handy, but the following paper addresses a similar, but stronger, version of the problem, i.e., if the distribution tails follow a power law.

POWER-LAW DISTRIBUTIONS IN EMPIRICAL DATA SIAM Review 51 (2009): 661--703.

$\endgroup$
1
$\begingroup$

One approach that had been suggested to me was via the Central Limit Theorem.

This is a old question, but I want to propose a way to use the CLT to test for large tails.

Let $X = \{X_1,\ldots,X_n\}$ be our sample. If the sample is a i.i.d. realization from a light tail distribution, then the CLT theorem holds. It follows that if $Y = \{Y_1,\ldots,Y_n\}$ is a bootstrap resample from $X$ then the distribution of:

$$Z = \sqrt{n}\times\frac{mean(Y) - mean(X)}{sd(Y)},$$

is also close to the N(0,1) distribution function.

Now all we have to do is perform a large number of bootstraps and compare the empirical distribution function of the observed Z's with the e.d.f. of a N(0,1). A natural way to make this comparison is the Kolmogorov–Smirnov test.

The following pictures illustrate the main idea. In both pictures each colored line is constructed from a i.i.d. realization of 1000 observations from the particular distribution, followed by a 200 bootstrap resamples of size 500 for the approximation of the Z ecdf. The black continuous line is the N(0,1) cdf.

enter image description here enter image description here

$\endgroup$
  • 1
    $\begingroup$ No amount of bootstrapping will get you anywhere against the problem I raised in my answer. That's because the vast majority of samples will not supply any evidence of a heavy tail--and bootstrapping, by definition, uses only the data from the sample itself. $\endgroup$ – whuber Apr 7 '17 at 21:35
  • 1
    $\begingroup$ @whuber If the X values are taken from a symmetrical power law, then the generalized CLT applies and KS test will detect the difference. I believe that your observation do not correctly characterize what you say is a "gradual step from "finite" to "infinite"" $\endgroup$ – Mur1lo Apr 7 '17 at 21:39
  • 1
    $\begingroup$ The CLT never "applies" to any finite sample. It's a theorem about a limit. $\endgroup$ – whuber Apr 7 '17 at 21:43
  • 1
    $\begingroup$ When I say that it "applies" I'm only saying that it provides a good approximation if we have a large sample. $\endgroup$ – Mur1lo Apr 7 '17 at 21:48
  • 1
    $\begingroup$ The vagueness of "good approximation" and "large" unfortunately fail to capture the logic of hypothesis tests. Implicit in your statement is the possibility of collecting an ever larger sample until you are able to detect the heavy-tailedness: but that's not how hypotheses tests usually work. In the standard setting you have a given sample and your task is to test whether it is from a distribution in the null hypothesis. In this case, bootstrapping won't do that any better than any more straightforward test. $\endgroup$ – whuber Apr 7 '17 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.