1
$\begingroup$

I would like to measure the accuracy of an algorithm at predicting a certain target value. Since there is some variance in the algorithm's output, I've run it a few times and computed the Pearson correlation between the predictions and the targets each time. Afterwards, I took the average of the correlations using Fisher z-transform.

Can I compute the p-value for this averaged correlation the standard way for Pearson correlations with t-distribution? (i.e. http://www.jeremymiles.co.uk/usingstatistics/chapter8/sigofrexcel.html) The number of samples is the same each time the algorithm is run (the target values are the same), so there does not appear to be any weighting required.

There were quite a few related questions, but none of them seemed to have the particular answer I was looking for:

p-value of an Average Correlation Coeficient

Averaging correlation values

Significance of average correlation coefficient

Improve this question
$\endgroup$
3
  • $\begingroup$ I think any theory you do find will be for the case of independent samples which is not your situation. $\endgroup$
    – mdewey
    Commented Dec 17, 2016 at 17:15
  • $\begingroup$ In most cases the Pearson correlation coefficient will be a poor choice of assessing the accuracy of the algorithm: it's too sensitive to a single outlying value, for instance. Use a more direct measure, such as a summary of the actual errors made. Regardless, a p-value is meaningless as a measure of accuracy. $\endgroup$
    – whuber ♦
    Commented Dec 17, 2016 at 17:50
  • $\begingroup$ Thanks @whuber. I certainly don't disagree that the significance is not particularly useful (a non-zero correlation does not mean good prediction) - I am also reporting RMS error. I've found that many clinical papers will report p-values so I decided to include them (with appropriate critique in the discussion section, of course). $\endgroup$
    – limi44
    Commented Dec 18, 2016 at 2:06

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.