P-values from non-normal distribution

Question

I have a collection (approximately 12,000) correlation values. Our correlation analysis does not allow for negative correlations (we correlate with sinusoidal waves, so instead of a negative correlation, we simply positively correlate with the opposite phase).

This produces a distribution of positive r values. Traditionally we have converted this distribution to z-scores using (in MATLAB):

zVal = sqrt(degreesOfFreedom)/2 * log((1+rVal) ./ (1-rVal));

Where sigma = sqrt(degreesOfFreedom) and z = 1/2 * log((1+rVal) ./ (1-rVal))

I'm pretty sure this is the "correct" calculation, although I am suspicious it is inappropriate because the r values are not normally distributed (fails the KS test, as well). Next, we convert to p-values using:

pVal = 1 - normcdf(abs(zVal), 0, 1); # Assumes normal distribution w. mean = 0, sigma = 1

My question is, will the above calculation give you incorrect p values if your r values are not drawn from a normal distribution? I'm running into some resistance from higher-ups, but I'm pretty sure that this is wrong.

This is what the data looks like:

Would I be right to say that 'normcdf' is responsible for the enormous number of highly significant items? Is there a way to directly identify p values from an arbitrary distribution? Due to the incredible skew in the distribution of p values, most multiple-comparison correction procedures produce rather liberal p-thresholds.

Answer 1

It looks like the problem is that your null hypothesis is on the boundary of your parameter space.

You are correct that under $H_0: r = \rho$ and appropriate regularity conditions, the test statistic, $z = \sqrt{df}/2 \left( \log\left(\frac{1+\hat{r}}{1-\hat{r}} \right) - \log\left(\frac{1+\rho}{1-\rho}\right) \right)$ is asymptotically normal. The problem is that among the appropriate regularity conditions is that $\rho$ is in the interior of the set of values that you might estimate for $\hat{r}$, i.e. the parameter space. Usually, the parameter space for a correlation is [-1,1], and testing $H_0: r=0$ is fine. However, it sounds like for your problem, the parameter space is [0,1]. In this case $z$ (with $\rho=0$) cannot have a normal distribution. I would guess your $z$ has $\chi^2$ distribution, but I have not worked it out.

Answer 2

This is intended to complement the nice answer from Paul, not contradict it.

The simplest theory for estimating correlation (indeed anything) is to postulate that there is one unknown constant value, and each sample provides an estimate of that.

There seems a kind of hint in your posting that correlation might vary any way. If that's true the usual kind of target is itself a distribution, and the right kind of treatment will vary too.

Regardless of that, the crucial thing is not so much how many correlations you have (although 12000 is good) but on what sample size each is based. As Paul did flag, the standard results are asymptotic, and the folklore seems to be that that's not to be understated: convergence is not fast.