0
$\begingroup$

In order to estimate predictive power of the model, we:

1)split the data on the training set(80%) and test set(20%).

2)Then we build the model based on training set

3)We use, for example, k-fold cross-validation in order to determine how good our model is.

Question 1 : I cannot understand why do we need to do first step, when in the third step(k-fold cross-validation) does the same procedure even better?

Question 2: Do we need to do something after third step? Our it is the last step of procedure? If so, why do we need to do first step?

$\endgroup$

1 Answer 1

0
$\begingroup$

We need the split the data into training and testing so that the ultimate test of the learned model is performed on data that was not used directly in its learning.

Cross validation is only used to test how making some changes such as different choices of some parameters affects the learned model and its generalization ability. But you do not have the luxury of tuning parameters after you observe, for example, a high error in the test set. Hence you have this intermediate cross validation step where you are allowed to see the validation error and tune the parameters until it is satisfactorily low. So in effect, the test error confirms that whatever changes you made based on the CV error, did indeed improve the model's generalizability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.