# Is it valid to take a mean of p-values during cross-validation, when comparing the predicted output of a model to the actual output? [closed]

I am doing a cross-validation study, training a model on an input to predict a target.

During training, my model generates an output vector that is guaranteed to be the same size as the corresponding training target vector. I can use metrics such as $R^2$ or RMS error to quantify this.

During testing, my model can produce an output vector that is not the same size as the input (but they are the same order of magnitude). I'm wondering if there are any ways to quantify the similarity between the model output and the testing set targets.

What I've come up with so far is to compare the distributions under the null hypothesis that the model output distribution is the same as the test target distribution. I'm using things like the Kolmogorov–Smirnov test, Ansari-Bradley test, or a permutation test. For each cross-validation fold, there is 1 p-value. Is it valid to report a mean of p-values to summarize this? Or are there better ways to do this?

• why don't you just compute the mean square error or misclassification error of the test data set for each CV fold and then average them? – matt Mar 11 '15 at 11:03
• This question makes no sense to me. How can your model "produce an output vector that is not the same size as the input"? Please explain this in detail. In the meantime, I vote to close as unclear. – amoeba Feb 15 '17 at 22:41
• @amoeba I think the cross-validation training fold size or test fold size are what he is talking about, they do not have to be equal sizes. – Carl Feb 16 '17 at 18:50
• @Carl This: During testing, my model can produce an output vector that is not the same size as the input -- does not occur in cross-validation. – amoeba Feb 17 '17 at 10:00