Skip to main content
Cross Validated

Return to Answer

deleted 1 character in body
Source Link
geekoverdose
geekoverdose
  • 3.9k
  • 2
  • 17
  • 28

This will depend on how your training and test sets are composed.

If the test set is rather big and reflects the "application case" data diversity correctly, I would not argue like this. But if the test data is rather small, you could of course achieve some good or bad results by chance. Using more test data would be helpful is such cases (or using a bigger portion of the total data available - if possible).

Further, training results should be obtained using some inner partitioning (e.g. repeated cross validation), which tests on data the modelsmodel has not seen before. The performance and performance spread across those results shows you how your model usually performs, and how likely it is to just obtain better or worse results. Using such a procedure, I would not consider any test results that are better than your CV results to be realistic. You should probably also look at and compare the CV performance and performance spread of both models.

And: keep in mind that if your training data is rather small compared to your test data, your training results might still be noticeably better than your test results and real application case results.

This will depend on how your training and test sets are composed.

If the test set is rather big and reflects the "application case" data diversity correctly, I would not argue like this. But if the test data is rather small, you could of course achieve some good or bad results by chance. Using more test data would be helpful is such cases (or using a bigger portion of the total data available - if possible).

Further, training results should be obtained using some inner partitioning (e.g. repeated cross validation), which tests on data the models has not seen before. The performance and performance spread across those results shows you how your model usually performs, and how likely it is to just obtain better or worse results. Using such a procedure, I would not consider any test results that are better than your CV results to be realistic. You should probably also look at and compare the CV performance and performance spread of both models.

And: keep in mind that if your training data is rather small compared to your test data, your training results might still be noticeably better than your test results and real application case results.

This will depend on how your training and test sets are composed.

If the test set is rather big and reflects the "application case" data diversity correctly, I would not argue like this. But if the test data is rather small, you could of course achieve some good or bad results by chance. Using more test data would be helpful is such cases (or using a bigger portion of the total data available - if possible).

Further, training results should be obtained using some inner partitioning (e.g. repeated cross validation), which tests on data the model has not seen before. The performance and performance spread across those results shows you how your model usually performs, and how likely it is to just obtain better or worse results. Using such a procedure, I would not consider any test results that are better than your CV results to be realistic. You should probably also look at and compare the CV performance and performance spread of both models.

And: keep in mind that if your training data is rather small compared to your test data, your training results might still be noticeably better than your test results and real application case results.

Source Link
geekoverdose
geekoverdose
  • 3.9k
  • 2
  • 17
  • 28

This will depend on how your training and test sets are composed.

If the test set is rather big and reflects the "application case" data diversity correctly, I would not argue like this. But if the test data is rather small, you could of course achieve some good or bad results by chance. Using more test data would be helpful is such cases (or using a bigger portion of the total data available - if possible).

Further, training results should be obtained using some inner partitioning (e.g. repeated cross validation), which tests on data the models has not seen before. The performance and performance spread across those results shows you how your model usually performs, and how likely it is to just obtain better or worse results. Using such a procedure, I would not consider any test results that are better than your CV results to be realistic. You should probably also look at and compare the CV performance and performance spread of both models.

And: keep in mind that if your training data is rather small compared to your test data, your training results might still be noticeably better than your test results and real application case results.