6
$\begingroup$

I'm computing a weighted correlation coefficient, using the method described here.

I'd like to compute a p-value for the resulting r coefficient. How can I do this correctly, given that my r was computed using weights? Naturally, the standard formula for p-value of r (e.g., here) does not take weights into account, and I'm not sure how to properly account for weights when computing the p-value.

$\endgroup$

2 Answers 2

6
$\begingroup$

The $P$-value reported for a correlation depends on the sample correlation, the sample size, and a bundle of assumptions not always checked (independence being, in my experience, least checked of all). But there is a difference between a crude $t$-based $P$-value based on a null hypothesis of zero correlation and a more general $P$-value based on Fisher's $z$ transformation.

I don't think there is an answer to this independent of what the weights are. If weighting means that you are combining data from different subsamples, then the weights have implications for the sample size that should be used; at the same time correlations based on weighted combinations would not necessarily have the same distribution as the correlation distribution based on raw data.

At the same time, it is difficult to get agitated about this. If correlations have a point it is that they measure strength of relationship; if you are seriously in doubt that they are significantly different from zero, then it is arguable that you just have inadequately small samples and being precise about that problem is secondary.

It's likely that this misreads your problem, in which case you may have to give much more detail.

If getting really reliable $P$-values for weighted correlations is important to you, it is possible that you need to get a handle on it through simulation, including simulation of the weighting process if that is variable too.

$\endgroup$
1
  • 2
    $\begingroup$ I think that's the right reading of my problem. Both (a) it seems like the answer depends on the weights, and (b) simulation seems like a last resort if there's no closed form. And your general philosophy makes sense. Still, I was hoping for a generic calculation (analogous to the t-based P-value based on a null hypothesis of zero), presumably in terms of the weights. $\endgroup$
    – gb7688
    Apr 21, 2014 at 18:23
0
$\begingroup$

I haven't yet sit down to work the math, but from the few simulations I run it seems that replacing the number of samples $n$ in the formulas with the effective number of samples $n_\text{eff}$ yields very good approximations.

$n_\text{eff} = \exp(H)$, where $H=-\sum_{i=1}^n w_i \ln w_i$ is the entropy of weights (normalized to $\sum_{i=1}^n w_i = 1$).

For example:

  • $t = r\sqrt{\frac{n_\text{eff}-2}{1-r^2}}$ follows approximately $t$-distribution with $n_\text{eff}-2$ d.o.f.,
  • $F(r) = \frac{1}{2}\ln\left(\frac{1+r}{1-r}\right)$ follows approximately normal distribution with mean $F(\rho)$ and std. $\frac{1}{\sqrt{n_\text{eff}-3}}$.

But the point should be made that even the unweighted formulas for $p$-values are approximations assuming normal data etc. Bootstrap or permutation tests might be more reliable and they work with Spearman's weighted correlation as well.

$\endgroup$
8
  • $\begingroup$ An extreme case helps to focus the mind. Suppose the correlation is based on two points distinct on both variables, each representing a group with a known frequency. Then the correlation necessarily has magnitude 1, and what you know -- that groups are almost always heterogeneous and so the underlying correlation will be (much?) less -- can't help in going beyond the spuriously high correlation. $\endgroup$
    – Nick Cox
    Feb 15, 2021 at 12:22
  • $\begingroup$ I suspect that the question that can't be studied easily -- why are we seeing weighted data and what lies beyond what we are given -- is often as or more important than e.g. whether each variable is or is not normally distributed. Bootstrapping or permutation can't help there, so far as I can see. $\endgroup$
    – Nick Cox
    Feb 15, 2021 at 12:23
  • $\begingroup$ @NickCox I'm sorry, I don't follow your example with two points. Obviously the sample correlation will be +/- 1, but the p-value should tell you, that this could very likely happen by chance. The formulas in my answer (t-test or z-test on Fisher transformation) are undefined for n=2 and bootstrap / permutation test should estimate the p-value to be very high. $\endgroup$ Feb 15, 2021 at 15:03
  • $\begingroup$ @NickCox The weights could have the interpretation as number of repeats, i.e. instead of repeating the pair $(x_i, y_i)$ $w_i$ times, you use $w_i$ as the weight. Or, that the weight is proportional to the inverse of the variance in estimating the point $(x_i, y_i)$. This way you can study the problem theoretically. $\endgroup$ Feb 15, 2021 at 15:06
  • $\begingroup$ @NickCox Bootstrap: you resample the triples $(x_i, y_i, w_i)$, for each you compute $r$, from the distribution of those you can compute confidence intervals or p-value for given $r$. Permutation test: this you can use when the weight $w_i$ is tied to one of the variables, e.g. $x_i$, but not the $y_i$. Then you can keep $(x_i, w_i)$ and permute $y_i$ to get the estimation of the null distribution and estimate the p-value as the quantile at 0. $\endgroup$ Feb 15, 2021 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.