Let's say we have two real valued data sets $x$ and $y$, both of length $n$. I do not want to make any further assumptions regarding these data sets. We're interested in their correlation. For testing whether it may be zero, we can run a permutation test. It should be enough to keep $x$ fixed and just permute $y$.
Now I wonder, is the expected value of the correlation in the permuted data actually zero? I suspect it is (and I have run a test with 100000 permutations for two pretty wild $x$ and $y$ and it was compatible with mean zero). But how to prove that? (Note that we are not in the situation of i.i.d. sampling from the empirical distribution here, as we draw without replacement.)