3
$\begingroup$

Given a matrix of observations (rows) x variables (columns), can we compute the correlation matrix of the rows, but corrected by the correlation matrix of the columns? The goal would be to avoid inflation in the correlation p values, since the t-test for Pearson correlation assumes that the columns/variables are uncorrelated, independent observations, i.e. that the correlation matrix of the columns is the identity matrix.

Intuitively, this could be accomplished with a weighted correlation, where e.g. if a pair of variables are nearly perfectly correlated, they would each be down-weighted by a factor of 2.

Edit: there is an exact null distribution for the correlations in the case where the columns are independent, but there are only two rows and this pair of rows is sampled from a bivariate normal distribution. I would like a null distribution for the opposite case, where the rows are independent, but the columns are sampled from a multivariate normal distribution.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Can you explain symbolically the assumption you're referring to (made by the t-test for Pearson correlation)? $\endgroup$
    – eric_kernfeld
    Commented Oct 30, 2017 at 12:43
  • $\begingroup$ @eric_kernfeld: The assumption is that $t = r \sqrt{\frac{n - 2}{1 - r^2}}$ is t-distributed with $n - 2$ degrees of freedom under the null, where $r$ is the correlation and $n$ is the number of data points. $\endgroup$
    – 1''
    Commented Oct 30, 2017 at 18:51
  • $\begingroup$ That's not what I meant to ask for. When you write "assumes that the columns/variables are independent", can you write out what that means to you and how it differs from the model you have in mind? $\endgroup$
    – eric_kernfeld
    Commented Oct 30, 2017 at 19:04
  • $\begingroup$ Sorry, that was a bit ambiguous - I edited the question. $\endgroup$
    – 1''
    Commented Oct 31, 2017 at 0:13

1 Answer 1

Reset to default
1
$\begingroup$

Here's something you can try.

Let $X$ be your data matrix. Normalize $X$ as you choose.

Calculate the $p \times p$ correlation matrix $M$ for the columns of $X$.

Then write the singular value decomposition of $M$ as $$USV^T = M.$$ Define a new data set $$Y = X U \sqrt{S} .$$

Then calculate the correlation matrix for the rows of $Y$.

This is in some sense what you are after: the "adjusted" correlation matrix for the rows of $X$.

Improve this answer
$\endgroup$
6
  • $\begingroup$ If I understand correctly, without the $\sqrt{S}$ this would be PCA, but the $\sqrt{S}$ additionally weights by the square root of the percent of variance explained by each component. One desirable property of a solution is that if you duplicate a column of the data matrix $X$ (say with a bit of noise), the correlation p values should stay the same. Here, I think you'd have the same top principal components, but the PCs most similar to the duplicated column would have larger eigenvalues, which would change the correlation p values. $\endgroup$
    – 1''
    Commented Nov 1, 2017 at 22:12
  • $\begingroup$ Not sure what the question here is. I provided a real example in the edit. Play around with it. $\endgroup$
    – Open Season
    Commented Nov 1, 2017 at 23:06
  • $\begingroup$ I was saying I think it's not robust because if you duplicate one of the columns, the p values change. I think all SVD-based solutions have this problem. $\endgroup$
    – 1''
    Commented Nov 1, 2017 at 23:43
  • $\begingroup$ I'm curious whether you could instead use $Y = XU$ (PCA) or $Y = XSU$ (PCA weighted by % variance explained) rather than $Y = X\sqrt{S}U$ (PCA weighted by sqrt of % variance explained)? $\endgroup$
    – 1''
    Commented Nov 1, 2017 at 23:45
  • $\begingroup$ @1'' Hey sorry I had a typo in the answer, the multiplication order should be switched. $\endgroup$
    – Open Season
    Commented Nov 2, 2017 at 0:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.