0
$\begingroup$

I can observe full system dynamics from some deterministic start to some deterministic end, in order to collect a sub-dataset $\boldsymbol{X}_i$ with $m$ examples, where the distribution of these $m$ examples may be highly imbalanced. I can do this $n$ times to produce a dataset $\boldsymbol{X}=\{X_1,\ldots,X_N\}$, with $n\times m$ examples.

Let's say I want to evaluate some algorithm on the dataset $X$ via K-fold cross validation.

My question is: is it better to randomly partition the total set of $n\times m$ examples into K folds, which may randomly result in training and validation sets that are rather unrepresentative of a full system observation, or should I randomly partition $n$, so my training sets and validation sets always contain some multiple of a full system observation?

The former seems more 'in line' with original k-fold cross validation, but the latter seems more principled, because I'm using some knowledge about the underlying system in order to ensure each model includes one or more complete samples of the population.

EDIT: I should say: my target is to predict some response for a subset of the $m$ examples from a testing sub-dataset.

$\endgroup$

1 Answer 1

0
$\begingroup$

The answer here depends on how you best simulate the population you will be generalizing towards post-deployment of your model.

If the population is balanced, then balance your test/validation folds. If you anticipate the population balance to be random, then randomly sample. If you expect it to be consistently imbalanced, then enforce an imbalance.

Short story: Your k-folds enable your validation, and the meaning behind your validation is tied to how well it represents the population, as that is what will dictate actual performance.

Caveat: If you expect significant heteroscedasticity of the balance over some dimension or metric, then sample across that metric or dimension accordingly, so that you get a realistic estimate of the population of balances you might see in the future.

Example: Time series forecasting is usually validated with folds taken over continuous periods of time to simulate seasonality, or time-correlated heteroscedasticity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.