0
$\begingroup$

A weird question below.

Suppose you did a 10 fold cross-validation to show that a model is an unbiased estimator. And the results of the cross-validation also shows that training the model longer will improve its prediction accuracy. However you do not have the computational capacity to keep training all 10 folds. Is it scientifically correct to refit the model with all the data and continue training to obtain the final model. Will the results from the 10-fold cv be invalid? Would you still be able to conclude that the model is unbiased?

cheers

$\endgroup$
2
$\begingroup$

Here are a few thoughts - hope this helps!

In my experience, I've used cross-validation primarily for assessing internal validity and parameter tuning, so one conventional way to use it is to identify the parameter of interest that you're tuning, optimize it over the 10-folds and re-train the full model on the final selected parameter setting. Once you've trained your final model, I'd hesitate to train further, because you're essentially departing from your "optimal" model setting that you obtained via a cross validation scheme.

For example, say you're applying an $L_1, L_2$ penalty on your model, with associated $\lambda_1, \lambda_2$ parameters dictating its magnitude, then you run the 10-fold CV to find the optimal $\lambda_1^*, \lambda_2^*$, and train the final model with them. If you continue training, then you might end up with different $\lambda^+$ values that are not validated via CV.

If number of epochs is another parameter you want to tune (i.e., want to test 50, 100, 200, etc..), then simply introduce it into your parameter grid search and select the best $n^*$ from your grid search. The one thing to be careful with training a model too long (specially with optimization schemes like gradient descent) is that eventually they'll start to overfit, so check the training and validation error with plots like the below. You basically train until your validation error stops improving (not the training error).

enter image description here

Here's the link to where I got that image (https://www.jeremyjordan.me/evaluating-a-machine-learning-model/). This person has a great tutorial on model evaluation.

I'll defer to others more knowledgeable on memory-optimization strategies - good luck!

$\endgroup$
1
$\begingroup$

I am not 100% sure what is your question,

but the principal cross-valididation works like this: enter image description here

so basically during every training and test split you will have different parts of the data. This is usefull to test if you are overfitting (biased) or underfitting your algorithm by get a specific training part of your data.

So you can also split your data step by step, when you do not have enough memory error and train and test it with the same algorithm, this is the same as running it in one run.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.