1
$\begingroup$

Say I have a data set.

I split the data into 10 chunks.

I fit the model on the 1st chunk, and test it on the 9 other chunks.

The average result over the 9 chunks is then the average performance of the model.

What is this cross-validation technique called?

Note that I am ONLY fitting the model ONCE (on the 1st of the 10 chunks). I am NOT refitting it on multiple chunks. So I am not sure what this is called .."1-fold cross validation"? I'm not sure.

$\endgroup$
5
  • 1
    $\begingroup$ How is evaluating on nine chunks and averaging different from gluing the nine together and scoring on one test set? At the moment, it doesn’t sound like there’s any ‘cross’ in this cross-validation. $\endgroup$ Apr 2, 2021 at 20:38
  • $\begingroup$ Why are you splitting your test set into chunks? And, what is your performance metric? $\endgroup$
    – gunes
    Apr 2, 2021 at 20:56
  • $\begingroup$ Well, say I have two models. If I test on all 9 chunks glued together at once, then model B might lose to model A if it performs poorly on one particular part of that total chunk. But if I split it up into 9 chunks, then I might see that model B "wins" 8 times out of 9, but just happens to do particularly badly at the 9th set. This is a more informative validation test than just testing it on the glued together chunks. $\endgroup$
    – Makei
    Apr 2, 2021 at 21:03
  • $\begingroup$ Can you clarify how you intend to split the data into the 10 chunks? $\endgroup$
    – B.Liu
    Apr 2, 2021 at 21:13
  • 2
    $\begingroup$ Consider that what you’re doing, in the limit, would be to take every sample in the combined test set, check whether A or B performed better, and aggregate that. That’s what accuracy already does, though. If your test set is fragmented into 9 parts, this tells you more about the specifics of those samples than about A and B. $\endgroup$ Apr 2, 2021 at 22:12

1 Answer 1

2
$\begingroup$

Turning my comments into an answer: there isn’t cross-validation in this strategy, and it’s not more informative about model performance than measuring accuracy after gluing the nine test sets together.

Well, say I have two models. If I test on all 9 chunks glued together at once, then model B might lose to model A if it performs poorly on one particular part of that total chunk. But if I split it up into 9 chunks, then I might see that model B "wins" 8 times out of 9, but just happens to do particularly badly at the 9th set. This is a more informative validation test than just testing it on the glued together chunks.

Consider that what you’re doing, in the limit, would be to take every sample in the combined test set, check whether A or B performed better, and aggregate that. That’s what accuracy already does, though. If your test set is fragmented into 9 parts, this tells you more about the specifics of those samples than about A and B.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.