I have been reading through questions here to find an answer to my question but didn't find a satisfactory one. Here is my scenario:

I have a system computing an algorithm on each of 100 units (so basically, the system will generate a 100 values each corresponds to one unit). I have two different algorithms to predict the value the system will generate for a given unit. I randomly generated 120 subsets of the 100 units. For a prediction algorithm, I computed the predictions of each subset then computed Pearson's correlation coefficient between actual and predicted values for each subset. Eventually, I got 120 correlations r for each of the prediction algorithms.

My questions are:

  1. Is it meaningful to compute the average correlation for each prediction algorithm over the 120 correlations? I am using Fisher's Z to compute the average as explained here. So can I safely say that for these 100 units, algorithm X managed to predict value by correlation of = average correlation?
  2. I need to compare the significance of difference in predictive power of the two algorithms in such scenario, how can I do this having 120 correlation values per algorithm? can I normally apply a t-test in such case?

Thanks in advance,

  • $\begingroup$ What do you mean by "the system will generate a 100 values each corresponds to one unit"? What do you mean when you say "I randomly generated 120 subsets of the 100 units"? Did you sample with replacement? With respect to your first question, why are you looking at correlations rather than some measure of closeness of the prediction to the data? With a correlation, you have have predictions that are wildly off, but still have a high correlation. (Think of predictions that are off my an order of magnitude but in the same rank order as the data.) $\endgroup$ – Joel W. Sep 1 '14 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.