Apologies for large post/question:
I presented 116 subjects with displays of frequency distributions for 2 groups of data and asked subjects to provide subjective ratings on a 10 point Likert scale to indicate how strongly they believed the displayed groups of data were significantly different from each other (called strength-of-evidence ratings).
There were 8 separate trials with the means of the groups (and thus the mean difference between the groups) and the SD of the groups varied between trials
I tested subjects at pretest, provided an educational intervention, and then tested them again at posttest.
I want to investigate if the statistical characteristics of the trials' displayed groups (i.e., the mean difference, SD, t-score for the group differences) are significantly and normativley associated with subjects strength-of-evidence ratings and see if the educational intervention had an impact.
I have calculated 3 pretest and 3 posttest Pearson's correlation coefficients, individually for each subject, between their strength-of-evidence ratings and (a) the mean difference between the groups, (b) the SD of the groups, and (c) the t-score for the difference between the groups (which represents the normative integration of both mean difference and SD).
I wanted to test: (i) if the average of subjects' correlations for a particular statistical characteristic differed from 0 (i.e., were subjects strength-of-evidence scores associated with the statistical characteristic being considered), and (ii) was there a significant difference between the average pretest and posttest correlation for each statistical characteristic.
Initially I had just performed (i) a standard one-sample t-test on the simple mathematical mean of subjects' correlations for each statistical characteristic to see if they each significantly differed from 0; and (ii) a standard paired-sample t-test on the simple mathematical mean for subjects' pretest and posttest correlations for each statistical characteristic, to see if they differed significantly.
A committee member stated that it is inappropriate to perform standard statistical procedures on a Pearsons correlation coefficient due to the properties of the distribution of correlations (i.e., skewed/non-normal distribution of correlations), and recommended that I use the Fisher r-to-z transformation and significance test instead.
I attempted to do this and ran into 2 problems.
Problem #1) I calculated the Fisher r-to-z transformation on each subjects' correlation coefficients and averaged all subjects' Fisher z scores together to get an average Fisher z score that I planned to perform the significance test on. However, I wanted to back-transform this Fisher z score to a correlation to see how similar this average correlation was to the simple arithmetic average of my Pearson correlations. I was shocked to find that all of my average correlations that had been back-transformed from the average Fisher z scores were between .05 and .07 larger than the simple arithmetic averages were. This was at least double or triple the size of the difference expected according to https://psycnet.apa.org/record/1987-14534-001 (the closest approximation from that article was for samples based on 10 datapoint pairs and population correlations between .3 to .6)
Question #1 - does this mean that Fisher r-to-z transformations are not appropriate with my data for some reason? for either calculating the average correlation or for significance testing? I feel that if I was to perform a significance test on this average Fisher z score I feel like it would be sort-of-meaningless as it actually corresponds to a back-transformed average correlation that is quite inflated (.05 to .07 larger than my actual, simple arithmetic average correlation)
Question #2 - Why might I be getting so much inflation in my average correlation when calculated by Fisher r-to-z transformation? Might this be due to the small number of data pairs (i.e., only 8) that my correlations are based on?
Problem #2) If I provisionally accept these average Fisher z scores and perform significance testing and 95% confidence interval building on them I seem to run into a really serious problem due to my small number of data pairs (i.e., 8) that my correlations were based on. For example, if I want to test if my average Fisher z score of .66 is significantly different from zero, I believe I need to divide value by a standard error for the Fisher Transformation - which for this procedure (analogous to a one-sample t-test) I believe it is calculated as 1/sqrt(N-3), where N = the # of data pairs that made up the initial correlation calculations. In my case, with 8 data point pairs that would = .45. Which seams incredibly huge!!! with this large of a standard error, my critical Z score would = 1.47 (i.e., .66/.45), with a non-significant p = .07. When I calculate the 95% confidence interval with this and back transform to a Pearsons correlation, I end up with: r=.58, p=.07, 95% CI [-.22, .91]!!!!!!! If I have the Standard Error formula correct, it seams nearly impossible to attain significance with this few data point pairs?
-Question #3: Is my standard error for the Fisher Z correct for this procedure? is it supposed to be nearly impossible to have a significant correlation with only 8 data point pairs, despite having 116 subjects?
-Question #4: If my standard error formula is correct (and if it is supposed to be nearly impossible to have a significant correlation with only 8 data point pairs), then is there another procedure that I could use instead of Fisher Z to appropriately calculate the significance of these correlation coefficients?