I have a collection (approximately 12,000) correlation values. Our correlation analysis does not allow for negative correlations (we correlate with sinusoidal waves, so instead of a negative correlation, we simply positively correlate with the opposite phase).
This produces a distribution of positive r values. Traditionally we have converted this distribution to z-scores using (in MATLAB):
zVal = sqrt(degreesOfFreedom)/2 * log((1+rVal) ./ (1-rVal));
sigma = sqrt(degreesOfFreedom) and
z = 1/2 * log((1+rVal) ./ (1-rVal))
I'm pretty sure this is the "correct" calculation, although I am suspicious it is inappropriate because the r values are not normally distributed (fails the KS test, as well). Next, we convert to p-values using:
pVal = 1 - normcdf(abs(zVal), 0, 1); # Assumes normal distribution w. mean = 0, sigma = 1
My question is, will the above calculation give you incorrect p values if your r values are not drawn from a normal distribution? I'm running into some resistance from higher-ups, but I'm pretty sure that this is wrong.
This is what the data looks like:
Would I be right to say that 'normcdf' is responsible for the enormous number of highly significant items? Is there a way to directly identify p values from an arbitrary distribution? Due to the incredible skew in the distribution of p values, most multiple-comparison correction procedures produce rather liberal p-thresholds.