Is there a way to compare model performance in the caret package by combining the resampled predictions from every fold/repeat? I am working with a small (<1000 rows) and severely imbalanced (4% positives) data set and I am primarily concerned with precision (using prSummary as the summaryFunction). When I use 10-fold 5-repeat cross validation, the individual precision-recall curves seem to be unstable and a poor representation of model performance. Is it possible to use all of the resampled predictions to build a single curve (rather then the average of 50 curves) and compare it to models with different tuning parameters?
This is my first question on Cross Validated/Stack Exchange so please let me know if there are any other details I can provide.