I am new to machine learning research, and I have a general question regarding how to compare different models on the same data set. How does a top researcher do this, and what is expected from a typical conference such as ICML?

For this reason, let us assume I followed the following research methodology/approach to get my results:

  1. I created a new dataset for my specific problem.
  2. I trained state-of-the-art neural networks on my dataset
  3. I developed a novel "layer" based on assumptions about relevant things for my specific problem. Other papers did not take my specific approach into account.
  4. I can get models with better performance on relevant metrics (Precision, Recall, F1) than the baseline methods on my data set. But the issue is: to reproducibly achieve models of superior performance, I have to restrict my training of the enhanced model more (stop training after x epochs, use higher regularization values, and so on). Now, I am wondering if this is fair? And how do I mention this in the right way? Doing the same things by the other model does not improve performance.

The biggest problem I see is in my methodological approach to proof point 4. And this is my problem: my dataset consists only of a training and test split. I basically adjusted hyperparameters on my approach and the standard models to get the best results on all of them on the test data. So I optimized against my test data. But I think, you usually do this with a third split, but my labelled data is not enough.

Now I am worried that a reviewer may assume that I have not properly tuned the hyperparameters for the standard approaches so that my newly developed approach looks better. How can I handle this point? And how could I avoid this assumption? It is theoretically impossible to test all possible hyperparameters. Therefore, I approximated the hyperparameters for the standard models based on fast training loss (at the beginning) and then looked which combination gives the best results on the test data. Now I want to report the best model in each case in the paper. Is this a sound and accepted procedure? Or do I need to change this or describe it differently?

Thanks for any suggestion. If you can give me examples or literature on this topic, please let me know. I am new to ML research and want to do it the right way, and what would be the best design for my research?

  • $\begingroup$ Of possible interest: Benavoli, Alessio, et al. "Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis." The Journal of Machine Learning Research 18.1 (2017): 2653-2688. jmlr.org/papers/volume18/16-305/16-305.pdf $\endgroup$
    – Dave
    Jan 18 at 23:42

Your proposed methodology is flat-out invalid if you don't use a train/test/validation split. The point of the holdout dataset (I'll call this the test split even though some call it the validation split) is that you tune your hyperparameters on the validation set and pick the best hyperparameters, and finally you evaluate a single model with chosen hyperparameters on that test set which has never been seen or used in this process.

By not having a holdout set, you're effectively allowing both your neural network and the other standard approaches to cheat in terms of data leakage from the test set. Even if your results are better, you can't dismiss the potential criticism that your method simply allows more data leakage and more cheating.

  • $\begingroup$ Thanks a lot for your reply. I will create a validation data set as a split from the training data and use it for hyperparameter tuning and model selection for the final results. Then I can avoid this aspect of critic. Are there any other recommendations? $\endgroup$ Jan 19 at 12:58
  • $\begingroup$ Do you have an opinion on my thought that I have not tested all possible hyperparameter combinations? $\endgroup$ Jan 19 at 12:59
  • $\begingroup$ In a paper, one cannot "prove" that they've given competing algorithms equal opportunity to succeed. There's limited space for that. What you can do is briefly mention the efforts you've taken. Common scenarios: only default hyperparameters for the standard model are used; both default and selected hyperparameters are used; a collaboration with the author of the model to ensure optimal selection; and no mention whatsoever of the efforts taken. Efforts don't prove the optimal procedure was used either, but again, this is typically minor within a paper even if it could reverse the conclusions. $\endgroup$
    – Vic
    Jan 19 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.