Apologies for large post/question:

I presented 116 subjects with displays of frequency distributions for 2 groups of data and asked subjects to provide subjective ratings on a 10 point Likert scale to indicate how strongly they believed the displayed groups of data were significantly different from each other (called strength-of-evidence ratings).

There were 8 separate trials with the means of the groups (and thus the mean difference between the groups) and the SD of the groups varied between trials

I tested subjects at pretest, provided an educational intervention, and then tested them again at posttest.

I want to investigate if the statistical characteristics of the trials' displayed groups (i.e., the mean difference, SD, t-score for the group differences) are significantly and normativley associated with subjects strength-of-evidence ratings and see if the educational intervention had an impact.

I have calculated 3 pretest and 3 posttest Pearson's correlation coefficients, individually for each subject, between their strength-of-evidence ratings and (a) the mean difference between the groups, (b) the SD of the groups, and (c) the t-score for the difference between the groups (which represents the normative integration of both mean difference and SD).

I wanted to test: (i) if the average of subjects' correlations for a particular statistical characteristic differed from 0 (i.e., were subjects strength-of-evidence scores associated with the statistical characteristic being considered), and (ii) was there a significant difference between the average pretest and posttest correlation for each statistical characteristic.

Initially I had just performed (i) a standard one-sample t-test on the simple mathematical mean of subjects' correlations for each statistical characteristic to see if they each significantly differed from 0; and (ii) a standard paired-sample t-test on the simple mathematical mean for subjects' pretest and posttest correlations for each statistical characteristic, to see if they differed significantly.

A committee member stated that it is inappropriate to perform standard statistical procedures on a Pearsons correlation coefficient due to the properties of the distribution of correlations (i.e., skewed/non-normal distribution of correlations), and recommended that I use the Fisher r-to-z transformation and significance test instead.

I attempted to do this and ran into 2 problems.

Problem #1) I calculated the Fisher r-to-z transformation on each subjects' correlation coefficients and averaged all subjects' Fisher z scores together to get an average Fisher z score that I planned to perform the significance test on. However, I wanted to back-transform this Fisher z score to a correlation to see how similar this average correlation was to the simple arithmetic average of my Pearson correlations. I was shocked to find that all of my average correlations that had been back-transformed from the average Fisher z scores were between .05 and .07 larger than the simple arithmetic averages were. This was at least double or triple the size of the difference expected according to https://psycnet.apa.org/record/1987-14534-001 (the closest approximation from that article was for samples based on 10 datapoint pairs and population correlations between .3 to .6)

  • Question #1 - does this mean that Fisher r-to-z transformations are not appropriate with my data for some reason? for either calculating the average correlation or for significance testing? I feel that if I was to perform a significance test on this average Fisher z score I feel like it would be sort-of-meaningless as it actually corresponds to a back-transformed average correlation that is quite inflated (.05 to .07 larger than my actual, simple arithmetic average correlation)

  • Question #2 - Why might I be getting so much inflation in my average correlation when calculated by Fisher r-to-z transformation? Might this be due to the small number of data pairs (i.e., only 8) that my correlations are based on?

Problem #2) If I provisionally accept these average Fisher z scores and perform significance testing and 95% confidence interval building on them I seem to run into a really serious problem due to my small number of data pairs (i.e., 8) that my correlations were based on. For example, if I want to test if my average Fisher z score of .66 is significantly different from zero, I believe I need to divide value by a standard error for the Fisher Transformation - which for this procedure (analogous to a one-sample t-test) I believe it is calculated as 1/sqrt(N-3), where N = the # of data pairs that made up the initial correlation calculations. In my case, with 8 data point pairs that would = .45. Which seams incredibly huge!!! with this large of a standard error, my critical Z score would = 1.47 (i.e., .66/.45), with a non-significant p = .07. When I calculate the 95% confidence interval with this and back transform to a Pearsons correlation, I end up with: r=.58, p=.07, 95% CI [-.22, .91]!!!!!!! If I have the Standard Error formula correct, it seams nearly impossible to attain significance with this few data point pairs?

-Question #3: Is my standard error for the Fisher Z correct for this procedure? is it supposed to be nearly impossible to have a significant correlation with only 8 data point pairs, despite having 116 subjects?

-Question #4: If my standard error formula is correct (and if it is supposed to be nearly impossible to have a significant correlation with only 8 data point pairs), then is there another procedure that I could use instead of Fisher Z to appropriately calculate the significance of these correlation coefficients?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.