1
$\begingroup$

I'm fairly new to DS scene and I have been learning about theories and doing practices on kaggle/participate in private competition.

For real world problems, my understanding is that you split out test set from what you have, use training set for modeling building and tuning hyperparameters (gridsearch), and use test set to find best model, then possibly deploy the model.

I'm getting little confused with regards to competitions that provide test set that has no answers, but provide public score upon submission.

For those cases, is it better to…

  1. use same procedure (creating own test set) OR
  2. use all data available to train, and use public score to pick best model.

I'm assuming max daily submission of 2-5 and public score comes from 50% of test set, and training set is ~1000, probably RF or XGboost models.

My main concern is whether using more training set and relying on public score to find better model is better for doing well on competitions. Published kernels usually show only one model and don't split out its own test set, so I wonder if such steps are performed outside published kernels. The presence of public score and it being called test set confuse me :(

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I'm not sure about what you call "test sample" but it depends on the kaggle.

Either you have only one data sample and you have to separate the sample in 3 ways (if enough data): train sample (to learn your model), validation sample (to tune hyperparameters) and test sample (to achieve final evaluation for metric chosen).

Or you have a train sample and a test sample, basically you use your train sample for train+validation split and use the test sample to be the final evaluation.

There is always a bit of misunderstood about the difference between validation and test sample.

It must be clear that the validation sample(s) are to tune hyperparameters for complex machine learning or chose the best model among different regression (reggresion 1 with x1 and x2, regression 2 with x1 only, etc.) for example.

The test sample is not mandatory if data is not enough but always welcome to have for final assessment and ensure yourself that your model is really well generalized.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Thanks, I think my confusion stems from solving real world problem vs kaggle. Let's say I use xgboost, so hyperparameter tuning is taken care of through k-fold. In real world problem, there is no test set out there, so I would need to split out test set initially to see how the model performs in the end. In kaggle, there is already test set (usually), so I wonder if it's beneficial or detrimental to split out my own test set from training set. If it's beneficial, how do I best utilize two test sets? $\endgroup$
    – bchoiNY
    Commented Jul 2, 2019 at 13:39
  • 1
    $\begingroup$ The test set for kaggle is just the one on which the final evaluation of the model is done. You don't need then to do a test sample on your own in this case, you just use train + validation for the best model and then assess its performance with the kaggle test sample. If you think it's not good enough, then you train again, validate again, test again. Normally, the performance on the validation set is always really really close of the test sample, it can be seen repetitive for some people even but in kaggle context it's just done for everyone to have the same test sample to evaluate the perf. $\endgroup$
    – josef_joestarr
    Commented Jul 2, 2019 at 19:15
  • 1
    $\begingroup$ Thanks! That's what I was trying to get confirmation on. $\endgroup$
    – bchoiNY
    Commented Jul 3, 2019 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.