I searched for this in many forums, but didn't find any answers that would answer my question (or I didn't understand correctly). So, I will post here.
- I have a dataset A
- I have a machine learning algorithm (e.g., SVM) with already defined hyperparameters, but on another dataset (e.g., dataset B). I can't change the defined hyperparameters, but I can retrain it.
- I have a machine learning algorithm (e.g., MLP) whose hyperparameters need to be tuned and for that I will use dataset A.
My goal is to tune the MLP hyperparameters and then compare which one has better performance, the already fine tuned SVM or the MLP that I tuned.
I can use the K-Fold Cross Validation to find the best choice of hyperparameters by evaluating which setting has the best perfomance (e.g., best arithmetic mean of accuracy) in test folds, at the end of cross validation. Right? And then, having the best hyperparameter configuration, if my goal was to build a model, I would simply train the MLP on all data of dataset A. Right? However, my goal so far is not to create a model but to compare the performance of selected MLP hyperparameters against SVM.
Therefore, having the best hyperparameter configuration, how can I compare the MLP performance against the SVM with the least possible bias? After all, the SVM has been tuned to dataset B, so if I run SVM in the same K-Fold Cross Validation process on dataset A, my comparison will be biased and favoring MLP, 'cause:
- I chose the best hyperparameter setting of MLP by the best average accuracy on dataset A.
- The SVM hyperparameters were adjusted on dataset B.
Finally, where am I going wrong and how can I compare the MLP against SVM using K-Fold Cross Validation?
PS: I can't use hypothesis testing, holdout or leave one out cross validation