1
$\begingroup$

I've got two basic questions for machine learning as newcomer.

  1. Cross-validation vs Test. I've heard people or online said it's expected that cross-validation data accuracy score is always higher than test data?

Is that true? To some extent it makes sense because there's always certain degree of overfit for training data. But my own experience is always two scores are similar, for example, 0.82 and 0.83

  1. What metrics should we look at for model evaluation? What I usually do is: Accuracy score for classification and RMSE for regression.

Should we do accuracy score for regression?

Thanks

$\endgroup$
  • $\begingroup$ Note that cross validation used for validation (i.e. the final test, not for model tuning or selection) tends to have a small pessimistic bias if done correctly (= splitting is actually independent). $\endgroup$ – cbeleites unhappy with SX Apr 27 '17 at 9:25
  • $\begingroup$ This site works better when you ask only one question per post. If you have two questions, it's better to ask them separately. $\endgroup$ – D.W. May 24 '17 at 16:43
4
$\begingroup$
  1. The purpose of using a cross-validation set is to choose an optimal set of model hyperparameters. Therefore there is a certain amount of "hyperparameter overfitting" which can occur on the cross-validation set, which will then fail to carry over to the test set, resulting in a lower classification accuracy in most cases.
  2. The metrics you suggest are good go-to's but it really depends on what it is that you're designing the model to do. Here is a good list of different metrics, along with explanations providing some intuition for when you might want to use them.
| cite | improve this answer | |
$\endgroup$
1
$\begingroup$
  1. You are right the training accuracy shall be higher. The purpose of cross validation is to prevent overfitting (to do validation before the prediction). If the accuracy score is so close, maybe it's because your model is already quite good.

  2. Accuracy is very popular metrics, keep in mind precision and recall, F1 score are also important when you have unbalanced data. For instance you want to predict cancer (which is only 1%), so if you predict all of them to be without cancer, you get an accuracy of 99%, but the recall is 0. In this case the accuracy doesn't make too much sense because you failed to predict the cancer. Since you don't want to miss any case of the "False negative", recall (true positive/(true positive + false negative) is very important metric in this case.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.