I have a set of SVM models that I am trying to compare. Each of the models is trained on a variation of the original data:
- The original data
- The original data using resampling scheme A
- The original data using resampling scheme B
When I train the models, I use grid search to estimate the SVM hyperparameter $C$ and I get different values under each scheme.
The overall hypothesis that I am trying to test is whether or not resampling scheme B yields a model that performs better than resampling method A.
Is simply comparing performance metrics between scheme B and A acceptable? Or should I be using the same hyperparameters for the models for both schemes?