4
$\begingroup$

How can I determine if experimental data comes from a distribution where the variance is undefined (e.g. the Cauchy distribution)?

I honestly have no idea how to attack this problem in a sensible way, but I imagine one could use random matrix theory to help you. For example, the eigenvalues of a (N,N) non-symmetric matrix form a pattern depending on the distribution. For a standard normal distribution, the eigenvalues on the complex plane form a beautiful unit circle:

enter image description here (here $N=200$ and the results of 16 random matrices are shown with a red dot indicating the largest absolute value. All results are scaled by $\sqrt{N}$)

For the Cauchy distribution the results are quite different:

enter image description here (same values as above, including the scaling)

So random matrix theory predicts a signal, though it may be an over-engineered solution.

Question: What statistical test can determine if a distribution has undefined variance?

$\endgroup$
  • 1
    $\begingroup$ Related: stats.stackexchange.com/questions/2504/test-for-finite-variance $\endgroup$ – Henry Jan 23 '16 at 8:22
  • 1
    $\begingroup$ Your question seems to be a duplicate of the above link, except for that you offer a candidate solution. In my opinion, you could delete your question but post its solution part as an answer (even though it is not complete, it still has a gem of an idea in it which may be worthwhile) to the above-linked question. $\endgroup$ – Richard Hardy Jan 23 '16 at 9:09
1
$\begingroup$

I don't think you can attack this question directly. I tested stock market data on a truncated normal, a log-normal and a truncated Cauchy density. In fact, it is none of the above, of course as it is a mixture distribution. Nonetheless, my argument was that the dominant density would by the Cauchy distribution and that I could factor out the other effects.

I then tested the data using Bayes factors. I couldn't use the posterior as I did not have an exhaustive list of all possible explanations. A Bayes factor, in my case, would have been: $$\frac{\Pr(\text{data}|\text{truncated Cauchy distribution})}{\Pr(\text{data}|\text{truncated Normal distribution})}.$$

I ruled out the normal distribution over the entire population of trades with odds against the normal in the region of $10^{8.6\text{ million}}:1.$ The log-normal was ruled out for two reasons. The first was that the likelihood wouldn't integrate to unity given the prior because it was hyperbolic. The second was that the likelihood maximized only as the scale parameter went to infinity, which implied that the log-normal would eventually become the uniform distribution.

Your other option would be to test against the various Pearson families. Such a choice is still incomplete, however, as the Pearson families do not include all distributions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.