Two basic machine learning questions

Question

I've got two basic questions for machine learning as newcomer.

1. Cross-validation vs Test. I've heard people or online said it's expected that cross-validation data accuracy score is always higher than test data?

Is that true? To some extent it makes sense because there's always certain degree of overfit for training data. But my own experience is always two scores are similar, for example, 0.82 and 0.83

2. What metrics should we look at for model evaluation? What I usually do is: Accuracy score for classification and RMSE for regression.

Should we do accuracy score for regression?

Thanks

Answer 1

1. The purpose of using a cross-validation set is to choose an optimal set of model hyperparameters. Therefore there is a certain amount of "hyperparameter overfitting" which can occur on the cross-validation set, which will then fail to carry over to the test set, resulting in a lower classification accuracy in most cases.
2. The metrics you suggest are good go-to's but it really depends on what it is that you're designing the model to do. Here is a good list of different metrics, along with explanations providing some intuition for when you might want to use them.

Answer 2

1. You are right the training accuracy shall be higher. The purpose of cross validation is to prevent overfitting (to do validation before the prediction). If the accuracy score is so close, maybe it's because your model is already quite good.

2. Accuracy is very popular metrics, keep in mind precision and recall, F1 score are also important when you have unbalanced data. For instance you want to predict cancer (which is only 1%), so if you predict all of them to be without cancer, you get an accuracy of 99%, but the recall is 0. In this case the accuracy doesn't make too much sense because you failed to predict the cancer. Since you don't want to miss any case of the "False negative", recall (true positive/(true positive + false negative) is very important metric in this case.