Questions about confidence interval and presenting metrics for classification problem A friend of mine and I are working on our bachelor's thesis and we're getting close to its completion. We have a few disagreements about how to present the result metrics for the experiment that we're performing.
The thesis involves detecting geographical ditches by classifying raster pixels on maps. We're using a random forest probability classifier and also doing some post processing work on the model output before producing a binary classification (ditch or non-ditch for each pixel). Due to the way we're doing post-processing (involving neighbouring pixels) we've decided to split the geographical area we're working with into 11 different zones of equal size, and perform something similar to 11-fold cross validation to evaluate the model. We train a model on 10 of the zones and evaluate on the one we didn't train on. We do this 11 times until all zones have been evaluation zones once.
Now to the problem: The zones have varying amount of ditch pixels in them. Some may have only 50 000 ditch pixels while others have close to 500 000. If I understand correctly, the standard way you evaluate metrics with K-fold cross validation is simply to take the result from each fold, and averaging it with the results from the other folds. In our case I would argue that this leads to potentially lying with the statistics, due to the zones being so different in their class balance. Say one zone with 50 000 ditch pixels receives a 30 % recall, and one with 500 000 ditch pixels receives 90 % recall, it wouldn't be very accurate to say that the overall recall is 60 %.
Instead, wouldn't it be better to add all the predictions together and take one big metric from all 11 experiments?
Another thing we've been thinking about is if it's also accurate to produce confidence intervals on the metrics from the 11 folds. Will the confidence intervals not also technically be inaccurate if the class balance differences between folds are large?
Thanks beforehand!
 A: There is a very nice "answer c - none of the above".
For imbalanced data, you re-sample up, to make it less imbalanced, then evaluate off of that.  If you are very brave, and very careful, then you can use technical knowledge to fuzz your up-sampling and improve the quality of your training data.  Don't do this with your test data because it will falsely skew your results, making it look falsely worse.
I don't see why you can't present what it gives.  Test it on each of the cases and present the results that it yields.  If they are different cases, then they are different cases.
Your measure of goodness is something you should determine before you run the training.  Ask "what indicates the job I want this to do" and then pick the one or several measures that show that.  If, at the end, one of the viable measures more clearly, in your technical judgment, shows fine differences in performance better than the others then you can use that, but be cautious about doing such a thing because it can look contrived and negatively impact the credibility of your work.
You can also make clusters of similar regions and aggregate performance there.
You can also aggregate it all into one batch and give an aggregate performance for that.
I think the individual examples have the opportunity to showcase the performance in a way that other students can learn from now, and that you can find useful later on in your professional work.
