1
$\begingroup$

As far as I know in practice the whole training set is usually split into training, validation and test[1] sets. Training set is used to train the model, validation to tune the parameters and test set to evaluate the model performance. Does the unknown data(for which we don't know the labels) have the specific name?

For example in kaggle competitions we are usually provided with train and test [2] set and we are required to make a prediction for the test set. But this one is different from [1]. Should I call train_test to [1] or something different to [2]?

P.S. Sometimes in the code, I also see "dev" set but I guess this is just for "development" purposes.

$\endgroup$
1
$\begingroup$

The test set is the one used to evaluate performance of the trained model. Unlike the train/dev (dev is validation), in the test you have labels only for evaluation purposes, but you cannot use these labels for training.

As far as I see, your two definitions [1,2] are the same: it is a data set, not used for training but only to evaluate the trained model, and the labels are there solely for the purpose of knowing your performance on unseen data - you cannot feed this knowledge back into the training system.

The division to train/dev is more artificial, since both datasets are yours for training with their labels, but dev is usually used to tune hyper-parameters whereas train is used for the intrinsic training.

$\endgroup$
  • $\begingroup$ Thanks, makes sense to call test to both :) About the dev - here it seems Karpathy uses dev to just test the code quickly. He has the X_val separately. Is dev usual name for validation split? $\endgroup$ – Scientist1642 Jun 16 '16 at 13:52
  • $\begingroup$ I've seen works with only a dev set, and works with only a validation set, and their purpose was the same. I cannot say much more, I think that practically as long as it's a labeled data use can use, it's part of the "big" training set, and that you can divide however you want. What's important is to evaluate your system at the end on an unseen data set (the test set). $\endgroup$ – yoki Jun 16 '16 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.