I can relate to the way you're thinking about it, and of your 2 ideas, I'd say the first is clearly preferable:  Fisher designed *Z* such that any computations such as averaging are better done with *Z* than with the (0, 1)-bounded *r*.  But in this situation a sig. test seems less helpful, and its results would be less clear, than in the usual case. If there is one.  For one thing, we wouldn't have taken into account how many values went into each subject's *r*, so it'd take some thinking to figure out the best df for the test.   Have you considered simply showing the results and letting people make up their mind that way?  E.g., in a dot plot:

![enter image description here][1]

Edit:  perhaps better would be a bubble plot in which bubble size represented the N that went into each correlation.

  [1]: https://i.sstatic.net/Yf2TZ.jpg