Suppose there are two variables, x and y, and we wish to calculate some test statistic (say the Pearson correlation), and estimate its significance. The pvalue is calculated by estimating the probability that a random permutation of x will have a higher test statistic than the original one. The sample space is the permutations of indices, therefore.

A different way to calculate a pvalue, would use a different sample space. Say that we take a random subset of the rows, and calculate a linear prediction from x to y (which, of course, is closely related to the correlation). We could check if the prediction on the holdout data has a positive r2 score. Repeating this for many random subsets, we could define the pvalue as the estimated probability that a linear prediction from x to y, would not be beneficial on holdout data. Here the sample space is the subsets of indices.

Is there a rule (or some intuition) for choosing the sample space for calculating the pvalue? Specificcally, suppose that the goal is predicting y, assessing the prediction through cross validated r2. Suppose further that the purpose of the pvalue is to decide (e.g., through the FDR), whether x is a good feature for predicting y. Since the natural sample space for cross-validation is a subset of the rows, then the pvalue of the features should be calculated through the same sample space, no?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.