# P-values from non-normal distribution

I have a collection (approximately 12,000) correlation values. Our correlation analysis does not allow for negative correlations (we correlate with sinusoidal waves, so instead of a negative correlation, we simply positively correlate with the opposite phase).

This produces a distribution of positive r values. Traditionally we have converted this distribution to z-scores using (in MATLAB):

zVal = sqrt(degreesOfFreedom)/2 * log((1+rVal) ./ (1-rVal));


Where sigma = sqrt(degreesOfFreedom) and z = 1/2 * log((1+rVal) ./ (1-rVal))

I'm pretty sure this is the "correct" calculation, although I am suspicious it is inappropriate because the r values are not normally distributed (fails the KS test, as well). Next, we convert to p-values using:

pVal = 1 - normcdf(abs(zVal), 0, 1); # Assumes normal distribution w. mean = 0, sigma = 1


My question is, will the above calculation give you incorrect p values if your r values are not drawn from a normal distribution? I'm running into some resistance from higher-ups, but I'm pretty sure that this is wrong.

This is what the data looks like: Would I be right to say that 'normcdf' is responsible for the enormous number of highly significant items? Is there a way to directly identify p values from an arbitrary distribution? Due to the incredible skew in the distribution of p values, most multiple-comparison correction procedures produce rather liberal p-thresholds.

• Note that you should only expect the $z$'s to be normally distributed and the p-values to be uniform if you know that your null hypothesis is true. In the data you have, the $z$'s can have any distribution. To check your procedure, you should really simulate data where you know the null hypothesis holds, apply your test, and check that resulting distribution of p-values is uniform.
– paul
May 27, 2013 at 18:46
• Thank you both for your answers. We have determined that the statistics derived from a null (random) distribution is best approximated by a weibull distribution (or similar). We retrieve a more reasonable distribution of p-values by using the cdf of this null weibull distribution.
– jdv
May 29, 2013 at 19:17

It looks like the problem is that your null hypothesis is on the boundary of your parameter space.

You are correct that under $H_0: r = \rho$ and appropriate regularity conditions, the test statistic, $z = \sqrt{df}/2 \left( \log\left(\frac{1+\hat{r}}{1-\hat{r}} \right) - \log\left(\frac{1+\rho}{1-\rho}\right) \right)$ is asymptotically normal. The problem is that among the appropriate regularity conditions is that $\rho$ is in the interior of the set of values that you might estimate for $\hat{r}$, i.e. the parameter space. Usually, the parameter space for a correlation is [-1,1], and testing $H_0: r=0$ is fine. However, it sounds like for your problem, the parameter space is [0,1]. In this case $z$ (with $\rho=0$) cannot have a normal distribution. I would guess your $z$ has $\chi^2$ distribution, but I have not worked it out.

• OK, so since this isn't a normal distribution, I should try using the CDF of some other distribution (I'll look at chi square), not the normal distribution? Is there some way to estimate the distribution of each data-set and use the "customized" cdf for each to calculate the p-values, or should I just find one distribution that is good enough to approximate all data-sets? Thanks!
– jdv
May 27, 2013 at 18:08
• Yes, there is likely some other asymptotic distribution. To figure out what it is, I would need to know more about your data and procedure. This is likely due to my ignorance, but I'm not sure what "we correlate with sinusoidal waves, so instead of a negative correlation, we simply positively correlate with the opposite phase" means.
– paul
May 27, 2013 at 18:38
• In terms of estimating the distribution of each data set, yes, something like that is possible. It is likely that subsampling is valid for your problem. It is less likely that the bootstrap (or at least the usual non parametric bootstrap) is valid for your problem. See Andrews (2000) for a similar example.
– paul
May 27, 2013 at 18:41

This is intended to complement the nice answer from Paul, not contradict it.

The simplest theory for estimating correlation (indeed anything) is to postulate that there is one unknown constant value, and each sample provides an estimate of that.

There seems a kind of hint in your posting that correlation might vary any way. If that's true the usual kind of target is itself a distribution, and the right kind of treatment will vary too.

Regardless of that, the crucial thing is not so much how many correlations you have (although 12000 is good) but on what sample size each is based. As Paul did flag, the standard results are asymptotic, and the folklore seems to be that that's not to be understated: convergence is not fast.