# How to choose hyperparameters for comparing different algorithms?

Assume the main task is to classify the data $X$ and it is done based on an optimization setting: $$\min_W f(X,W)+\lambda g(X,W)$$ in which $W$ is the classifier parameter matrix.

Now consider that i want to compare my approach to 2 other methods from the literature and the difference between all 3 methods is in the function $g(X,W)$ which was used. Let's say i claim to found a better function $g(X,W)$ yielding better classification.

My question is how to choose the $\lambda$ parameter to have a fair comparison between the 3 approaches?

1- Should i choose an individual $\lambda$ for each method based on a 10-fold Cross Validation? In that case, should i mention each choice of parameter for each approach and each dataset separately?!

2- Should i choose one specific $\lambda$ for all 3 approaches? or the reviewers argue that it is not a fair way?

• Since $g$ is different, it stands to reason that the best $\lambda$ for the three methods will be different as well. What you are interested in is comparing the best parameterization of the various classifiers, because that's what would be used in practice, not a parameterization that was inferior due to it being partially derived from some other, unused, method's parameterization. Jan 25, 2018 at 20:12
• @jbowman: Then for each approach and each dataset i'd have different $\lambda$ parameter chosen, in that case is it expected to mention all these parameter values for different approaches and different datasets?
– Bob
Jan 25, 2018 at 20:20
• I don't know that you'd need to provide the specific parameter values, that would depend on what the journal expects, but at least you'd have to tell them that your methodology had chosen individual $\lambda$ for each method and each data set. Jan 25, 2018 at 20:23