What is the right machine learning research methodology to compare two neural network approaches against eachother?

Question

I am new to machine learning research, and I have a general question regarding how to compare different models on the same data set. How does a top researcher do this, and what is expected from a typical conference such as ICML?

For this reason, let us assume I followed the following research methodology/approach to get my results:

1. I created a new dataset for my specific problem.
2. I trained state-of-the-art neural networks on my dataset
3. I developed a novel "layer" based on assumptions about relevant things for my specific problem. Other papers did not take my specific approach into account.
4. I can get models with better performance on relevant metrics (Precision, Recall, F1) than the baseline methods on my data set. But the issue is: to reproducibly achieve models of superior performance, I have to restrict my training of the enhanced model more (stop training after x epochs, use higher regularization values, and so on). Now, I am wondering if this is fair? And how do I mention this in the right way? Doing the same things by the other model does not improve performance.

The biggest problem I see is in my methodological approach to proof point 4. And this is my problem: my dataset consists only of a training and test split. I basically adjusted hyperparameters on my approach and the standard models to get the best results on all of them on the test data. So I optimized against my test data. But I think, you usually do this with a third split, but my labelled data is not enough.

Now I am worried that a reviewer may assume that I have not properly tuned the hyperparameters for the standard approaches so that my newly developed approach looks better. How can I handle this point? And how could I avoid this assumption? It is theoretically impossible to test all possible hyperparameters. Therefore, I approximated the hyperparameters for the standard models based on fast training loss (at the beginning) and then looked which combination gives the best results on the test data. Now I want to report the best model in each case in the paper. Is this a sound and accepted procedure? Or do I need to change this or describe it differently?

Thanks for any suggestion. If you can give me examples or literature on this topic, please let me know. I am new to ML research and want to do it the right way, and what would be the best design for my research?

Answer 1

Your proposed methodology is flat-out invalid if you don't use a train/test/validation split. The point of the holdout dataset (I'll call this the test split even though some call it the validation split) is that you tune your hyperparameters on the validation set and pick the best hyperparameters, and finally you evaluate a single model with chosen hyperparameters on that test set which has never been seen or used in this process.

By not having a holdout set, you're effectively allowing both your neural network and the other standard approaches to cheat in terms of data leakage from the test set. Even if your results are better, you can't dismiss the potential criticism that your method simply allows more data leakage and more cheating.