I am doing a cross-validation study, training a model on an input to predict a target.
During training, my model generates an output vector that is guaranteed to be the same size as the corresponding training target vector. I can use metrics such as $R^2$ or RMS error to quantify this.
During testing, my model can produce an output vector that is not the same size as the input (but they are the same order of magnitude). I'm wondering if there are any ways to quantify the similarity between the model output and the testing set targets.
What I've come up with so far is to compare the distributions under the null hypothesis that the model output distribution is the same as the test target distribution. I'm using things like the Kolmogorov–Smirnov test, Ansari-Bradley test, or a permutation test. For each cross-validation fold, there is 1 p-value. Is it valid to report a mean of p-values to summarize this? Or are there better ways to do this?