Let $Y=\left[Y_1, Y_2, \dots, Y_n\right]$ be a sequence of exchangeable, approximately normal random variables. Let $X_1=\left[X_{11}, X_{12}, \dots, X_{1n}\right]$ and $X_2=\left[X_{21}, X_{22}, \dots, X_{2n}\right]$ be two predictions for the $Y$, i.e. $(X_{1i}, Y_i)$ and $(X_{2i}, Y_i)$ are ordered prediction-response pairs.

Using a permutation test, how should I assess if $X_1$ or $X_2$ is a better predictor of $Y$?

To test the strength of the correlation for a single predictor, $X_j$, in Permutation, Parametric, and Bootstrap Tests of Hypotheses (Good 2010), the sum of the cross products test statistic, $$ S = \sum X_{1i}Y_i $$ is proven to be the uniformly most powerful against normally distributed alternatives among all unbiased tests of the hypothesis that pairs of observations are uncorrelated.

To compare $X_1$ against $X_2$, I consider the test statistic $T$, $$ T = \sum X_{1i}Y_i - \sum X_{2i}Y_i = \sum (X_{1i} - X_{2i})Y_i $$ and preform permutation testing by comparing the observed value, $T\rightarrow t$, to the distribution of $T$ over all permutations of the predictor index $j$ for each response $Y_i$, (i.e. randomly swapping $X_{1i} \leftrightarrow X_{2j}$).

Is $T$ an admissible test statistic? Are there other test statistics I should consider? Are these test statistics related to measures of correlation? How sensitive is $T$ or another test statistic to the assumptions of normality of $Y_i$. Are there further literature sources I should consider besides (Good 2010)?

For additional context, this from structural biology: to assess computational predictions of experimentally measured change in free energy of due to mutation ($\Delta\Delta G$), where $n$ is on the order of a thousand. See for example, (Kellogg 2011); especially figure 2.

  • $\begingroup$ Look into Steiger's version of Williams' modifiation of Hotelling's test. $\endgroup$ – Ray Koopman Aug 9 '13 at 14:41
  • $\begingroup$ @RayKoopman Thanks. It looks like r.test in the R psych package computes this. It seems like Steiger's observation about using using Fisher's Z is important. Also Hittner, May, & Silver (2003) and Wilcox & Tian (2008), compare various correlated correlation coefficient tests. $\endgroup$ – momeara Aug 9 '13 at 20:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.