Between the two, should I use a model's training or validation threshold to get best results (from a distributed random forest binary classifier built using
h2o.ai) (especially when their values differ by orders of magnitude)?
h2o flow UI to create a RDF model and examined the testing and validation thresholds optimized for maximizing the minimum per class accuracy in each set (since the classes were imbalanced), eg. for the validation set ...
Now I have a set of new data that I would like to make a prediction on using the exported model. The RDF is just going to output probability values for both of the response classes, so I need to choose a threshold (in my case, for the
max_min_per_class_accuracy) to use as a cutoff point to delineate them.
Don't know much about machine learning, but to me, I would think that using the validation threshold would be better since the threshold that worked best with the training set would be too optimistic and overfit. However, the default threshold used by the
h2o python API is for the training set which makes me question my own preference (as again, I have little experience). So which should I use, the threshold for the training set or for the validation set? Thanks
Is this a wrong-headed question to be asking in the first place or is there something about this question that shows some general confusion about concepts? Let me know if so.