Given a matrix of observations (rows) x variables (columns), can we compute the correlation matrix of the rows, but corrected by the correlation matrix of the columns? The goal would be to avoid inflation in the correlation p values, since the t-test for Pearson correlation assumes that the columns/variables are uncorrelated, independent observations, i.e. that the correlation matrix of the columns is the identity matrix.

Intuitively, this could be accomplished with a weighted correlation, where e.g. if a pair of variables are nearly perfectly correlated, they would each be down-weighted by a factor of 2.

Edit: there is an exact null distribution for the correlations in the case where the columns are independent, but there are only two rows and this pair of rows is sampled from a bivariate normal distribution. I would like a null distribution for the opposite case, where the rows are independent, but the columns are sampled from a multivariate normal distribution.

  • 1
    $\begingroup$ Can you explain symbolically the assumption you're referring to (made by the t-test for Pearson correlation)? $\endgroup$ Oct 30 '17 at 12:43
  • $\begingroup$ @eric_kernfeld: The assumption is that $t = r \sqrt{\frac{n - 2}{1 - r^2}}$ is t-distributed with $n - 2$ degrees of freedom under the null, where $r$ is the correlation and $n$ is the number of data points. $\endgroup$
    – 1''
    Oct 30 '17 at 18:51
  • $\begingroup$ That's not what I meant to ask for. When you write "assumes that the columns/variables are independent", can you write out what that means to you and how it differs from the model you have in mind? $\endgroup$ Oct 30 '17 at 19:04
  • $\begingroup$ Sorry, that was a bit ambiguous - I edited the question. $\endgroup$
    – 1''
    Oct 31 '17 at 0:13

Here's something you can try.

Let $X$ be your data matrix. Normalize $X$ as you choose.

Calculate the $p \times p$ correlation matrix $M$ for the columns of $X$.

Then write the singular value decomposition of $M$ as $$USV^T = M.$$ Define a new data set $$Y = X U \sqrt{S} .$$

Then calculate the correlation matrix for the rows of $Y$.

This is in some sense what you are after: the "adjusted" correlation matrix for the rows of $X$.

  • $\begingroup$ If I understand correctly, without the $\sqrt{S}$ this would be PCA, but the $\sqrt{S}$ additionally weights by the square root of the percent of variance explained by each component. One desirable property of a solution is that if you duplicate a column of the data matrix $X$ (say with a bit of noise), the correlation p values should stay the same. Here, I think you'd have the same top principal components, but the PCs most similar to the duplicated column would have larger eigenvalues, which would change the correlation p values. $\endgroup$
    – 1''
    Nov 1 '17 at 22:12
  • $\begingroup$ Not sure what the question here is. I provided a real example in the edit. Play around with it. $\endgroup$ Nov 1 '17 at 23:06
  • $\begingroup$ I was saying I think it's not robust because if you duplicate one of the columns, the p values change. I think all SVD-based solutions have this problem. $\endgroup$
    – 1''
    Nov 1 '17 at 23:43
  • $\begingroup$ I'm curious whether you could instead use $Y = XU$ (PCA) or $Y = XSU$ (PCA weighted by % variance explained) rather than $Y = X\sqrt{S}U$ (PCA weighted by sqrt of % variance explained)? $\endgroup$
    – 1''
    Nov 1 '17 at 23:45
  • $\begingroup$ @1'' Hey sorry I had a typo in the answer, the multiplication order should be switched. $\endgroup$ Nov 2 '17 at 0:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.