# Training threshold vs validation threshold for better prediction results?

Between the two, should I use a model's training or validation threshold to get best results (from a distributed random forest binary classifier built using h2o.ai) (especially when their values differ by orders of magnitude)?

Details:

Used h2o flow UI to create a RDF model and examined the testing and validation thresholds optimized for maximizing the minimum per class accuracy in each set (since the classes were imbalanced), eg. for the validation set ... Now I have a set of new data that I would like to make a prediction on using the exported model. The RDF is just going to output probability values for both of the response classes, so I need to choose a threshold (in my case, for the max_min_per_class_accuracy) to use as a cutoff point to delineate them.

Don't know much about machine learning, but to me, I would think that using the validation threshold would be better since the threshold that worked best with the training set would be too optimistic and overfit. However, the default threshold used by the h2o python API is for the training set which makes me question my own preference (as again, I have little experience). So which should I use, the threshold for the training set or for the validation set? Thanks

• For my particular example, the training and validation thresholds that maximize the metric I'm using are in totally different ballparks (differing by orders of magnitude (eg. train_thresh=0.101, valid_thresh=0.000188)) (note that the classes in the data were very imbalanced and I used the balance classes flag in h2o flow UI to try to account for this). Would the same averaging logic hold in this scenario? – lampShadesDrifter Mar 9 '18 at 22:36