1
$\begingroup$

Cross-validation is used for estimation of model.

I misunderstand the concept. If different part of corpus is used for training per each iteration, then each iteration generates the different model. So, what's validated? Only selected features?

$\endgroup$
0
$\begingroup$

Cross-validation is a model validation method for assessing how a model will generalize to an independent data set. In a classification or regression problem, usually there is a dataset of known data on which training is run and a dataset of unknown data against which the model is tested. During the procedure of cross-validation a dataset is defined for testing the model in the training phase. Performing cross-validation technique we are able to notice how the model will generalize to an independent dataset and then tackle the issue of overfitting.

Overfitting is the phenomenon when a model is too complex and corresponds too closely or exactly to a particular data set and because of that may fail to fit additional data or predict future observations reliably". An overfitted model is a statistical model that contains more parameters than can be justified by the available data.

As a conclusion, using cross-validation you are validating the model and its parameters. It is not a way of making an estimation of the parameters.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Cross validation or more precisely: resampling validation relies on some assumptions. Relevant for your question are:

  • A (surrogate) model trained on the whole data set minus a few cases (the left-out fold) is approximately equivalent to the model trained on the whole data set (the one for which you want to measure generalization error).
    This assumption allows using the observed generalization performance of the surrogate models as approximation for the generalization performance of the model trained on the whole data set. Due to the lower number of training cases, you may have on average somewhat lower performance of the surrogate models: this is the well-known slightly pessimistic bias of resampling validation.

  • A second, weaker assumption is that the surrogate models are equivalent to each other.
    This allows sensible pooling of the observed results for all surrogate models and this assumption is violated if your models are not stable. So your point about "each iteration [fold] generates a different model" is rather seen the other way round: if the generated surrogate models are actually different, you do have a problem with model instability and anyways should go back and adapt your modeling strategy - and checking this should be part of your cross validation.

So once you checked that the surrogate models are reasonably similar, you are fine using their performance as approximation for the whole-data model's performance.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Cross-validation is used for estimation of model.

Not really. Cross-validation is used to validate the model. You can estimate any model for any data (obviously if they are supposed to be used for that type of data); in estimation, usually you compare models of the same family, but with different parameters, and you choose the best model of that family, by selecting the appropriate set of parameters. But once you have estimated the model, you have to validate it. That means three things: to measure how good is your model, in absolute terms; to check if the residuals are random (if it is a regression) and to check if your model is as good in the data in which you trained it as in another dataset (i.e. prevent overfitting).

Cross-validation is a non-parametric way of doing the latter. In deed in each iteration the model is different, so you are validating the fitting procedure as a whole. Ideally you need a hyper-parameter that regulates how complex is your model (e.g. the number of parameters of your model). Then you measure the out-of-sample accuracy by cross-validation, as a function of this complexity parameter, and you choose the value that minimize the error.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.