I am doing a cross-validation study, training a model on an input to predict a target.

During training, my model generates an output vector that is guaranteed to be the same size as the corresponding training target vector. I can use metrics such as $R^2$ or RMS error to quantify this.

During testing, my model can produce an output vector that is not the same size as the input (but they are the same order of magnitude). I'm wondering if there are any ways to quantify the similarity between the model output and the testing set targets.

What I've come up with so far is to compare the distributions under the null hypothesis that the model output distribution is the same as the test target distribution. I'm using things like the Kolmogorov–Smirnov test, Ansari-Bradley test, or a permutation test. For each cross-validation fold, there is 1 p-value. Is it valid to report a mean of p-values to summarize this? Or are there better ways to do this?

  • 3
    $\begingroup$ why don't you just compute the mean square error or misclassification error of the test data set for each CV fold and then average them? $\endgroup$ – matt Mar 11 '15 at 11:03
  • 2
    $\begingroup$ This question makes no sense to me. How can your model "produce an output vector that is not the same size as the input"? Please explain this in detail. In the meantime, I vote to close as unclear. $\endgroup$ – amoeba Feb 15 '17 at 22:41
  • $\begingroup$ @amoeba I think the cross-validation training fold size or test fold size are what he is talking about, they do not have to be equal sizes. $\endgroup$ – Carl Feb 16 '17 at 18:50
  • $\begingroup$ @Carl This: During testing, my model can produce an output vector that is not the same size as the input -- does not occur in cross-validation. $\endgroup$ – amoeba Feb 17 '17 at 10:00

How about using Fisher's method or Stouffer's method for combining independent p-values to reject the global null hypothesis ? In your case, I reckon the global null hypothesis is the training and test data follows the same distribution. For more information, you can visit the following pages

When combining p-values, why not just averaging?

Combining multiple p-values

| cite | improve this answer | |
  • 1
    $\begingroup$ Generally answers are better when they are standalone. Here you're referring to two webpages, and the links may or may not become broken at some point. You could consider expanding your answer to make it standalone without needing to refer to external websites. $\endgroup$ – Patrick Coulombe Oct 20 '15 at 3:11
  • $\begingroup$ These are not applicable because the different p-values do not come from independent tests! The cross-validation training sets have large overlap. Put differently: If you naively apply Fisher's method you'll get smaller combined p-value the more cross-validation folds you have, as the individual p-values are multiplied in the combined test statistic formula. $\endgroup$ – vbraun Jun 17 '17 at 16:09

Not the answer you're looking for? Browse other questions tagged or ask your own question.