I have been reading through questions here to find an answer to my question but didn't find a satisfactory one. Here is my scenario:
I have a system computing an algorithm on each of 100 units (so basically, the system will generate a 100 values each corresponds to one unit). I have two different algorithms to predict the value the system will generate for a given unit. I randomly generated 120 subsets of the 100 units. For a prediction algorithm, I computed the predictions of each subset then computed Pearson's correlation coefficient between actual and predicted values for each subset. Eventually, I got 120 correlations r for each of the prediction algorithms.
My questions are:
- Is it meaningful to compute the average correlation for each prediction algorithm over the 120 correlations? I am using Fisher's Z to compute the average as explained here. So can I safely say that for these 100 units, algorithm X managed to predict value by correlation of = average correlation?
- I need to compare the significance of difference in predictive power of the two algorithms in such scenario, how can I do this having 120 correlation values per algorithm? can I normally apply a t-test in such case?
Thanks in advance,