2
$\begingroup$

I'm trying to determine the best way to check if a sample of angles is uniformally distributed. As such I've built two toy functions to see what type of sampling error to expect from actually uniform distributions using scipy.stats and numpy.random.

import scipy.stats as stat
import numpy as np

def chiTest(n = 400, d = 10):
    theta = np.random.rand(n) * np.pi - np.pi/2 
    #theta should be uniform on [-pi/2, pi/2]
    bins = np.linspace(-np.pi/2, np.pi/2, n//d + 1)
    d, _ = np.histogram(theta, bins)
    return stat.chisquare(d)[-1]

Chi-squared gives me a p-value that is all over the place:

chiTest()
Out[297]: 0.41045635844052125
Out[298]: 0.78687422600627477
Out[299]: 0.016802707521273268
Out[300]: 0.66269332328844976

While Kolmogorov-Smirnov

def ksTest(n = 400):
    theta = np.random.rand(n) * np.pi - np.pi/2
    uni = np.linspace(-np.pi/2, np.pi/2, n)
    return stat.ks_2samp(theta, uni)[-1]

Is a bit better, but still gives false negatives

ksTest()
Out[350]: 0.31152830907597184
Out[351]: 0.93696634517876642
Out[352]: 0.56898463914998687
Out[353]: 0.74776878262002455

And gets even worse when my "control" distribution is actually random:

def ksTest2(n = 400):
    theta = np.random.rand(n) * np.pi - np.pi/2
    uni = np.random.rand(n) * np.pi - np.pi/2
    return stat.ks_2samp(theta, uni)[-1]

ksTest2()
Out[373]: 0.061328687673192099
Out[374]: 0.68866738727690713
Out[375]: 0.93696634517876642
Out[376]: 0.27118717204504289

The strange thing is that increasing n only seems to help ksTest(), but even big (1000000) sample sizes still can often throw p-values < 0.1. Am I misusing, incorrectly implementing, or just plain misunderstanding these methods? Or is this the best accuracy I can hope for considering the possibility of random clustering?

$\endgroup$
10
  • $\begingroup$ Regardless of sample size, a test of the null hypothesis is supposed to give p-values "all over the place"! They themselves should be uniformly distributed. A chi-squared test is an excellent choice for testing uniformity of angles: just make sure you don't let the software choose the bins. You should select them based on no knowledge of the data (apart from their count, to make sure the bin counts are large enough to make the chi-squared distribution applicable). $\endgroup$
    – whuber
    Aug 17, 2017 at 13:40
  • $\begingroup$ I'm sorry, I just can't wrap my head around how chi-squared can be an excellent test of uniformity, when it is also supposed to yield random p-values when testing for uniformity. $\endgroup$
    – Daniel F
    Aug 17, 2017 at 14:01
  • $\begingroup$ That is how all hypothesis tests work. Under the null hypothesis, $p$ is uniformly distributed. (I fudge a little, because there are exceptions for discrete distributions--but they don't change the basic idea.) Under alternative hypotheses, if your procedure is any good then $p$ will be concentrated towards smaller values. If that's not completely clear, then please study hypothesis testing before proceeding. Here's a good set of links: stats.stackexchange.com/…. $\endgroup$
    – whuber
    Aug 17, 2017 at 14:14
  • 1
    $\begingroup$ You seem to be confusing uniformity of the data at hand and uniformity of the p-value under the null hypothesis. p-values are uniform under the null essentially by construction. This is a mathematical fact, not something to be tested for. $\endgroup$
    – klumbard
    Aug 17, 2017 at 17:56
  • 1
    $\begingroup$ Well, pretty much. But that's not a feature of your specific question, that is how any hypothesis test works. We don't "accept" the null hypothesis if p > threshold, we can only say our observed data are consistent with the null hypothesis (i.e. "fail to reject"). I haven't looked at what you're doing specifically mostly because I'm too lazy to parse the Python code. Hence why I'm in the comments and not answering the question. If you fail to reject the null but don't like that we're not "accepting" it, then look into effect sizes for goodness of fit tests to quantify things more explicitly. $\endgroup$
    – klumbard
    Aug 18, 2017 at 13:19

2 Answers 2

2
$\begingroup$

Use a Kuiper test. It's location shift invariance makes it particularly suited to testing circular distributions.

$\endgroup$
0
$\begingroup$

My current workaround is to fit the data to a von Mises distribution, and if the k-value is suitably small, assume that the distribution is uniform.

def vmTest(n=400):
    theta = np.random.rand(n) * np.pi - np.pi/2
    k, t, _ = stat.vonmises.fit(theta * 2, fscale = 1)
    return k

This rules out a single bias direction, but not, for example, two orthogonal biases.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.