I'm fairly new to DS scene and I have been learning about theories and doing practices on kaggle/participate in private competition.
For real world problems, my understanding is that you split out test set from what you have, use training set for modeling building and tuning hyperparameters (gridsearch), and use test set to find best model, then possibly deploy the model.
I'm getting little confused with regards to competitions that provide test set that has no answers, but provide public score upon submission.
For those cases, is it better to…
- use same procedure (creating own test set) OR
- use all data available to train, and use public score to pick best model.
I'm assuming max daily submission of 2-5 and public score comes from 50% of test set, and training set is ~1000, probably RF or XGboost models.
My main concern is whether using more training set and relying on public score to find better model is better for doing well on competitions. Published kernels usually show only one model and don't split out its own test set, so I wonder if such steps are performed outside published kernels. The presence of public score and it being called test set confuse me :(