2
$\begingroup$

I'm trying to argue that my model is very robust.

Given are $N$-many x-y pairs as samples. Iterating over all samples, for each sample a model is trained using only this single sample. This model is then used to predict all other samples. For each prediction, the error is recorded (producing $N-1$ error values). In total, this creates $N \cdot (N-1)$ errors. All these errors are then aggregated by RMSE. Since the result is close to the RMSE of using all data to train my model, I call the model robust.

For lack of a better word, I've described this as leave-one-in cross validation. Is this a known idea?

I realize this is an unusual situation. For clarification, an example. Assume there are 5 samples numbered 1 to 5.

The model is trained with sample 1. The error of the model on samples 2 to 5 is calculated. Then a new model is trained with sample 2, and its error on samples 1, 3, 4 and 5 is calculated. Same procedure for the other 3 samples, each gets its own model.

Then, all errors are squared, averaged and sqrted, so there is an RMSE. This RMSE is only a little worse than a LOO CV RMSE or a fully trained model (little difference).

Improve this question
$\endgroup$
14
  • 1
    $\begingroup$ you may be interested in "One-shot learning" <en.wikipedia.org/wiki/One-shot_learning_(computer_vision)>, though this describes a learning task rather than a predictive error estimation strategy. $\endgroup$
    – John Madden
    Commented Apr 26 at 0:02
  • $\begingroup$ "Since the result is close to the RMSE of using all data to train my model..." In what way? Most reasonable models tend to have better RMSEs as the training sample size increases. But you seem to be training each model on a sample of size 1. None of these models (neither individually, nor on average) are likely to have RMSEs close to the RMSEs of training models on samples of size N. $\endgroup$
    – civilstat
    Commented Apr 26 at 0:06
  • $\begingroup$ I would assume the exact opposite, that this will lead to a very unstable model. What happens if you hold out a validation set and evaluate on that? Alternatively, what does the variance of RMSE look like under bootstrapping? $\endgroup$
    – Randy Savage
    Commented Apr 26 at 2:03
  • $\begingroup$ @civilstat Exactly. The validation RMSE of my model, trained with just a single sample, is similar to the LOOCV RMSE. This is unusual, which is why I say the model is robust. $\endgroup$
    – mafu
    Commented Apr 26 at 2:25
  • $\begingroup$ @RandySavage I'm not sure if I understand you correctly, but the validation set for each model is the entire remainder ($N-1$) of the samples. $\endgroup$
    – mafu
    Commented Apr 26 at 2:28

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.