I am not sure whether this is the right place to ask this question, so feel free to redirect me if not.

What I'm doing is bench-marking a model (MobileNet v2 100 224) in terms of performance - size of training sample, learning rate, image distortions and augmentation is various forms, basically trying to find the right hyperparameterization that works for out specific computer vision task. Now the question is whether the runs should use the same sample dataset and only change the parameters or should there be a separate sample for each of the runs. To be clear - I am performing 8 runs per one change in one parameter and then taking the average output (Overall accuracy, Precision, Recall, etc) and comparing between.


You should use the same training/testing splits for all hyperparameters to be able to compare them—otherwise you cannot know if the potential improvement comes from the hyperparameter combination or a "lucky data split".

Ideally, you could do cross-validation, using several different train/test splits, to avoid overfitting the hyperparameters to one particular test split. However, this might be computationally infeasible.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.