7
$\begingroup$

I am doing a cross-validation study, training a model on an input to predict a target.

During training, my model generates an output vector that is guaranteed to be the same size as the corresponding training target vector. I can use metrics such as $R^2$ or RMS error to quantify this.

During testing, my model can produce an output vector that is not the same size as the input (but they are the same order of magnitude). I'm wondering if there are any ways to quantify the similarity between the model output and the testing set targets.

What I've come up with so far is to compare the distributions under the null hypothesis that the model output distribution is the same as the test target distribution. I'm using things like the Kolmogorov–Smirnov test, Ansari-Bradley test, or a permutation test. For each cross-validation fold, there is 1 p-value. Is it valid to report a mean of p-values to summarize this? Or are there better ways to do this?

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ why don't you just compute the mean square error or misclassification error of the test data set for each CV fold and then average them? $\endgroup$
    – matt
    Mar 11, 2015 at 11:03
  • 2
    $\begingroup$ This question makes no sense to me. How can your model "produce an output vector that is not the same size as the input"? Please explain this in detail. In the meantime, I vote to close as unclear. $\endgroup$
    – amoeba
    Feb 15, 2017 at 22:41
  • $\begingroup$ @amoeba I think the cross-validation training fold size or test fold size are what he is talking about, they do not have to be equal sizes. $\endgroup$
    – Carl
    Feb 16, 2017 at 18:50
  • $\begingroup$ @Carl This: During testing, my model can produce an output vector that is not the same size as the input -- does not occur in cross-validation. $\endgroup$
    – amoeba
    Feb 17, 2017 at 10:00

1 Answer 1

Reset to default
1
$\begingroup$

How about using Fisher's method or Stouffer's method for combining independent p-values to reject the global null hypothesis ? In your case, I reckon the global null hypothesis is the training and test data follows the same distribution. For more information, you can visit the following pages

When combining p-values, why not just averaging?

Combining multiple p-values

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Generally answers are better when they are standalone. Here you're referring to two webpages, and the links may or may not become broken at some point. You could consider expanding your answer to make it standalone without needing to refer to external websites. $\endgroup$
    – Patrick Coulombe
    Oct 20, 2015 at 3:11
  • 1
    $\begingroup$ These are not applicable because the different p-values do not come from independent tests! The cross-validation training sets have large overlap. Put differently: If you naively apply Fisher's method you'll get smaller combined p-value the more cross-validation folds you have, as the individual p-values are multiplied in the combined test statistic formula. $\endgroup$
    – vbraun
    Jun 17, 2017 at 16:09
  • $\begingroup$ The testing sets are separate for what it's worth. It does feel like collecting all those p-value's might be useful though. If all of the p-value show significance that feels more meaningful than significance from one hold out set. I'm not really sure what you can do apart from take the minimum p-value and then get an underpowered test. Naively I think I would prefer the mean of the p-value to than the p-value from one holdout set. $\endgroup$
    – Att Righ
    Jan 5, 2022 at 10:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.