4
$\begingroup$

Suppose we have training data set and a test data set. The outcome variable is binary. Is it usually necessary to split the training data set so that there is a cross validation data set? Or can you use the whole training data set to build a model and the use this model on the test data set? For logistic regression, for example, would cross validation really help? If so, what type would be best?

$\endgroup$
5
$\begingroup$

Cross validation has two purposes :

  • when you don't use cross validation and randomly select a part of data as train and other part as test, you may have a high accuracy in that part for train and test but when you select another train and test data you may have lower accuracy. Cross validation methods like n-fold cross validation or etc. will help to find best fit model based on your database. with lowest error on all parts of data.

  • In some cases cross validation will help to find some parameters of model like C in logistic regression that you can find some documentation about it in MATLAB help center or in R documentation files.

So as we discoursed cross validation has a critical rule to find a reliable model for your database. You should select best cross-validation technique based on your model structure and your sample size. 5-fold cross validation is a well known technique. You can increase the k in k-fold cross validation If you have more sample size.

$\endgroup$
1
$\begingroup$

In general cross validation is always needed when you need to determine the optimal parameters of the model, for logistic regression this would be the $C$ parameter.

As a first start you can look into k-fold validation, if you are using R look at (http://caret.r-forge.r-project.org/training.html) or python (http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation)

$\endgroup$
  • 6
    $\begingroup$ The goal of CV is not to estimate parameters but to estimate the generalization performance and stability of your full learning procedure. If you choose to determine parameters with CV (you probably mean with a grid search), you should add another layer of cross validation, to cross-validate the grid search process itself. In other words, the full learning procedure, regardless of whether you are estimating parameters or hyper-parameters, should be cross validated. Please see the links I provided in the comments to the OP. $\endgroup$ – Amelio Vazquez-Reina Jul 16 '14 at 18:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.