I have an experiment in which I present a subject with $n$ inputs, $\pmb{x} \in \mathbb{R}^N$. For each input, a response is produced in ~25,000 separate output variables - so for a given output variable $Y_i$, $Y_i \in \mathbb{R}^N$.
For each $Y_i, i \in [0, 25000]$ and a function $f$ that maps inputs to features, I need to determine whether a linear regression model can be used to predict $Y_i$ given $f(\pmb{x})$, and if so, calculate the accuracy of this prediction. Prediction accuracy is defined by Pearson's r between predicted output $\hat{Y}_i$ and true output $Y_i$.
The method given in a paper for this is:
For each $Y_i$:
- Split $\pmb{x}$ into $\pmb{x}_{train}$ and $\pmb{x}_{test}$.
- Use k-fold cross validation on $\pmb{x}_{train}$ to determine whether the linear regression model predicts $Y_i$ from $f(\pmb{x}_{train})$ significantly better than chance, using a p threshold of 0.01 / 25000 = 4e-6 (to correct for the number of output variables).
- If the linear regression model was found to predict better than chance, then calculate the prediction accuracy by training on the entire training set and evaluating on the test set.
My issue is with the details of step 2. I understand k-fold cross validation but I don't know what test I should be using to determine whether the prediction is better than chance from the results of the cross validation. The paper's exact wording is: "Student’s t test across cross-validated [input]", but I don't know exactly what that means here.
Clarifications
Context: We have a set of output variables $S = \{Y_i, \forall i \in [0, M]\}$. We suspect that there is a subset of these $Q = \{Y_j \in S, \text{$Y_j$ can be predicted from $f(\pmb{x})$}\} \subseteq S$, where each $Y_j$ can be predicted using a linear mapping from $f(\pmb{x})$: in other words, where a better-than-chance correlation exists between $Y_j$ and our predicted values $\hat{Y}_j$.
Once $Q$ is determined, I can calculate the prediction accuracy of the variables by computing the Pearson's r between $Y_j$ and $\hat{Y}_j$ for all $Y_j \in Q$. This is in a biological context and the idea is that we can go onto analyse (for example) the biological reasons why only the elements in $Q$ can be modelled in this way, as opposed to all the elements in $S$.
The linear regression model being used is ridge regression, which is 'trained' by fitting it to training data then evaluating it on test data. Hyperparameter search is performed by repeating the cross-validation in step 2 over the grid of hyperparameters.
To use the wording from the paper, it determines Q by "...discarding output variables whose prediction accuracy was not significantly better than chance, $p > 0.01 / M$ (Bonferroni correcting for number of output variables), Student’s t test across cross-validated training inputs". This is too vague for me to understand. I did not think you needed to correct the p threshold for the hyperparameter search in ridge regression.
An idea we had: for all (k = 5) folds, compute Pearson's r between the predicted and true values for that fold, thus ending up with a sample of 5 r values. Use a one-sample t test to determine whether the mean of these 5 r values is different from 0, using $p=0.01/M$ as the threshold. If the null hypothesis is rejected, then this output variable is included in Q. However, I'm not sure that this method is valid.