I would like to seek expert advice on the topic above.
I was taught to follow this workflow:
- Split dataset into training and testing
- Use training dataset to develop model
- Set hyperparameter in model
- Do cross-validation (multiple train-validation splits, tests, measures)
- Repeat with different hyperparameter set until performance of train is near to validation
- Thereafter, check performance of final chosen model on testing dataset
My thoughts are:
- Testing dataset is chosen once
- It might not be representative
- Hence performance might not be representative also, be it good or bad
My questions are:
- Given this situation, would result from cross-validation be good gauge of model performance?
- If this is the case, then train-test split might not be needed
- Of course during cross-validation, it is important not to have data leakage
I would love to hear your thoughts. Thank you very much.