# Model comparison across multiple correlated responses

I have two multivariate linear regression models (multiple outcomes, i.e., the responses are a matrix), and I'm measuring their performance using $R^2$ in cross-validation, over these individual responses, and across all responses. I'd also like to quantify the difference in $R^2$ using a p-value, for example t-test (after Fisher transform of $R^2$ for better normality etc).

Now, these responses are highly correlated with each other. Therefore, averaging the $R^2$ over the responses into a single estimate worries me as they are not independent, and a significance test that ignores this might be badly biased. Should I instead get a separate p-value for each outcome and then use some multiple-testing correction?

-
(+1) Good question. If I understand this correctly, you have two multivariate regressions -- $\mathbf{Y} = \mathbf{X}_j\boldsymbol{\beta}_j + \boldsymbol{\varepsilon}_j, \,j=1,2$ and $\mathbf{Y}$ is $n\times l$, $\mathbf{X}_j$ is $n \times k_j$, and $\boldsymbol{\beta}_j$ are $k_j\times l$? That is basically, you have two competing regressor sets to explain your multiple outcomes? This typically leads to $l$ $R^2$ statistics per regressor set, and there is no accepted way of producing one scalar GoF statistic. Some advice is here. – tchakravarty Nov 29 '12 at 14:34
I'm happy to accept this as an answer if you post it as an answer. – purple51 Nov 29 '12 at 21:58