I have a set of SVM models that I am trying to compare. Each of the models is trained on a variation of the original data:

  • The original data
  • The original data using resampling scheme A
  • The original data using resampling scheme B

When I train the models, I use grid search to estimate the SVM hyperparameter $C$ and I get different values under each scheme.

The overall hypothesis that I am trying to test is whether or not resampling scheme B yields a model that performs better than resampling method A.

Is simply comparing performance metrics between scheme B and A acceptable? Or should I be using the same hyperparameters for the models for both schemes?


Indeed resampling methods can affect the performance of a model. This is particularly true when the data is imbalanced.

There is no reason to think the $C$ that you chose should be "universal" and I am not surprised that the best $C$ parameter depends on the sampling scheme you use to fit the model.

I assume that you used cross validation to evaluate the performance of each scheme, so you have to be very careful with the standard deviation of the performances obtained.

A last point to stress is the fact that, the more models you test, the more likely you are to find a model whose performance surpasses all the other models, out of pure luck.

With all this in mind, I would feel comfortable using a grid ($C$, sampling scheme) that does not contains too many elements.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.