Train-validation-test split for small and unbalanced dataset?

Question

I have a dataset of around 100 rows, each with around 400 features. 93 of them are class 0, and 7 are class 1.

I want to be able to split my 100 examples into a train set, a validation set, and a test set so that I can optimize hyperparameters and pick a good model.

However, my number of class 1 rows is so low that the way they get shuffled into validation or test set causes huge fluctuations in performance metrics.

What is the best way to split my dataset up in order to both pick good hyperparameters and report meaningful performance metrics?

Answer 1

A few thoughts in addition to Stephan Kolassa's and Christian Henning's answers:

• As Christian says, nested cross validation is the only sensible splitting scheme in small sample size situations. It helps to get most out of your sample, but it cannot work miracles: you still only have 93 and 7 cases, respectively.

  Specifically, I'd recommend: to use stratified repeated k-fold CV for both of the nested splits (or leave-few-out) and explicitly recommend against leave-one-out: in LOO, model instability and case-to-case variance (small finite sample) are collinear.

  The repetitions of the CV allow you to measure (separate) model instability variance from variance due to the limited test sample size, see e.g. our paper Beleites, C. & Salzer, R. Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 2008, 390, 1261-1271

• Consider whether a one-class classifier (aka class model) for the majority class is a sensible option. That will allow you to model the better known class without its boundary suffering from the extreme uncertainty due to having very few cases from the other class.

  You can then still evaluate the resulting class model also with the cases not belonging to that class.

  In general: iff a classification problem is truly binary, a binary model will be more efficient, i.e., fewer cases are needed to establish the class boundary compared to one-class classifiers.
  OTOH, the boundary of a binary classifier can become very uncertain if one class is ill-defined or too few cases are available.

  Whether it helps in your case needs to be determined, but IMHO it is worth a try.

  Update: since some comment says this is a medical problem, binary classification (or multiple mutually exclusive classes) corresponds to differential diagnosis, i.e. situations where you have the knowledge that none but the considered classes can occur, and they are mutually exclusive. One-class classification corresponds to diagnoses where potentially not all other possibilities are known or modeled, and/or classes are not mutually exclusive. (Someone with hepatitis may have some other infection as well. Or a tumor, etc.)

• While I fully agree with Stephan that you really need to use proper scoring rules, there are 1 - 2 points here where the hard evaluation metrics are useful: you can do back-of-the-envelope calculations with them beforehand. So, let's consider sensitivity for the 2 classes, and look at 95 % confidence interval width. That is narrowest for observed 100 % (or 0 %) sensitivity, and widest for 50 %.

  • For the majority class,
    observed 100 % -> c.i. is ≈96 - 100 %
    observed 50 % -> c.i. is 40 - 60 %
  • For the minority class,
    observed 100 % -> c.i. is 60 - 100 %
    observed 50 % -> c.i. is 20 - 80 % (yeah, that won't happen, but the point is: you cannot distinguish 50 : 50 guessing from apparent "perfect" prediction based on 7 tested cases.)

  The conclusions are:

  • If a confidence interval width of about half the possible value range for the minority class is unacceptable => you can stop working right now.

  • You will not be able to do any meaningful optimization including the minority class evaluation. Optimization of a one-class classifier can likely stop early as well, since there is also substantial variance uncertainty.

  • While proper scoring rules often have (besides other advantages) also better variance properties than the hard proportions, overfitting is likely a massive issue with n : p < 1/4. And overfitting often comes with the symptom of the models' predictions becoming "oversure", i.e. harder than they should be: overfit models often predict 0 or 100 % class membership rather than anything in between. And in that case, the proper scoring rules have as bad variance as the ("artificially") hardened proportions.

  So, use (strictly) proper scoring rules, but do not expect them to be subject to less variance uncertainty here.

• From a stats point of view, data-driven model optimization is often a massive multiple-comparison situation. Here, with non-negligible noise on the evaluation of the target functional.

  Even on the majority class, you can hardly afford much data-driven model optimization. Thus, include as much external knowledge into your modeling, and reduce (or refrain from) model optimization.

  An example of this from my field, chemometric modeling of spectroscopic data, is that we can spot some cases of overfitting by inspecting coefficient patterns for linear models for noise between neighbouring variates, and we know for reasons of physics and chemistry that (bi)linear models should typically be suitable. Thus, if I spot such noisy patterns already in the first (few) principal components, there is no need to consider any more complex models with the given data and I can restrict the search space for hyperparameters accordingly. (That model may still be overfit, which I can measure by repeated CV.)

---

Update: CV terminology

• stratification (here) means splitting in a way that preserves as closely as possible the relative frequencies of the classes.

• nesting means that a subset of cases is split off for evaluating/verifying/"validating" the final model, and then the remaining data (training set in the wider sense, i.e. including hyperparameter optimization) is split a 2nd time into a training set in the narrower sense (on which tentative models are fit using varying sets of hyperparameters) and a data set for "intermediate" estimation of predictive performance. These performance estimates are used as target for hyperparameter optimization/selection.

  Both splits need to yield statistically independent data subsets, other than that one can choose among a number of suitable "splitting procedures" (k-fold CV, single split, out-of-bootstrap).

So, nesting is needed in order to do data-driven hyperparameter optimization. If you fix all hyperparameters beforehand, you don't need the optimization and thus only one level of splitting (the one corresponding to the outer split of the nested setup).

• cross validation is a splitting procedure that produces a specified number of subset splits, $k$. Each case in the original data appears exactly once in all the $k$ test subsets (and is training case in all other of these splits), and the test subsets are to be as similar as possible in size.

  The surrogate models trained from the $k$ training subsets are assumed to be equivalent, and also to be a suitable approximation to the model trained on the whole data set. Thus, the results can be pooled and used as approximation to estimate the predictive performance of the model trained on the whole data set. This pooling is crucial in small sample size situations since it yields the maximal possible number of tested cases.

• $k$-fold CV is a variant of CV where the desired number of subsets is directly specified.
  (As opposed to leave-$n$-out, where the size of the test subsets is directly specified, and $k$ is a consequence of the total number of available cases divided by the desired subset size)

• repeated CV repeats the CV procedure in order to evaluate more surrogate models. This is possible for varieties of CV where the exact assignment of case -> test subset has a random component, i.e. k-fold or leave-n-out with more than 1 case per test subset.
  Since then the same case is tested with multiple different surrogate models, we can more easily separate the results into variance due to model (training) instability vs. case-to-case variation.

So I do a k-fold (for example, 5 folds), and then within my train set which consists of 80 of the samples, I do another k-fold to split into 64 and 16 samples?

yes

And then in the 64 samples that then make up my train set, I try sets of hyperparameters(i.e. GridSearchCV),

yes

then evaluate against 16 samples to find the best hypreparameters,

no: you evaluate against 5x16 = 64 samples, choose the optimal hyperparameters

Then train a model on the 80 cases with that hyperparameter set. Let's call the result the auto-tuned model.

then evaluate against my 20 sample test set?

again no: you evaluate 5 autotuned models (on 80 training cases) against 20 cases each, and thus get results for 100 test cases.

You train with your autotuning procedure an autotuned model on 100 training cases and use the 100 case test result as approximation for that model's performance.

Also is a one-class classifier mostly supposed to only identify outliers? In my case, my class with 93 samples is "healthy", so it wouldn't help much there.

More precisely speaking it's supposed to identify cases belonging to the class.

The healthy (normal!?) class model should then tell you for the other cases that they do not belong to the healthy class.

The important point is that from 93 healthy cases it may be possible to estimate better (less uncertain) class boundaries than from 7 "not-healthy" cases, even if they come from a well defined class (which not-healthy does not even need to be).

You may try to set up an additional whatever-disease-it-is class for the 7 other cases (if they can sensibly counted as belonging into the same class) - but this will have huge uncertainty on the class boundaries.

A discriminative/binary classifier with at most only 7 cases in one class will also have huge uncertainty in the class boundary due to only the 7 cases of the minority class.

Answer 2

However, my number of class 1 rows is so low that the way they get shuffled into validation or test set causes huge fluctuations in performance metrics.

Here is another way of looking at things: your target class is quite rare, so rare that you may not see it even in production in medium-sized samples. Consider a medical doctor who may not see a semi-rare disease in a whole year, or in their whole career. This does not mean that they should not know about this particular disease, but that their priors should be stacked against it.

Unfortunately, there is not very much you can do here. You simply have very little data. And a hugely complex model - 400 predictors for just 100 observations is far too much. No matter how much regularization and "model selection" you do, you will overfit. (I used scare quotes here because in a situation like this, you won't be selecting models, you will be chasing noise.) You can't magically conjure up information where there is none. Your only way forward to be able to build a decent model is to collect more observations.

Until then, I would recommend you pare down your 400 predictors to the ones that have the strongest connection to your outcome on a theoretical basis. Or possibly combine your predictors using PCA or similar.

And finally, of course use good evaluation metrics, not accuracy, precision, recall, F1 or anything else that uses hard 0-1 classifications. These all have major issues for "unbalanced" data (and also elsewhere). See also Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?. Rather, use probabilistic predictions and evaluate these using proper scoring rules. With the rarity in your data, your model might be tempted to output flat zero predictions, avoid this by using the log score rather than the Brier score: Why is LogLoss preferred over other proper scoring rules?. And finally, once you have your predictions, think hard about any comparison thresholds and set them in the context of your actual cost structure: Reduce Classification Probability Threshold

Answer 3

A useful way to quantify the difficulty of the task is to compute the effective sample size as discussed here. Here the ESS is $3np(1-p)$ where $p=0.07$; ESS=19.5. With that amount of information you might afford to analyze a single pre-specified predictor.

Answer 4

I agree with Stephan Kolassa generally. If you have only 7 observations from one class, the basis for hyperparameter selection and performance assessment is severely limited. Your data only carries a certain amount of information, and no method can go beyond that.

In any case one approach would be double (or nested) leave one out cross-validation (LOO-CV), i.e., running an outer LOO-CV to do performance measurement where you do the hyperparameter and model selection by an inner LOO-CV where each observation is left out one at a time for each data set generated by the outer CV. So in each run you have only two observations left out. (At the end you can use a single LOO-CV to pick the hyperparameters and select a model, but double LOO-CV to assess the performance.)

There's also double bootstrap by the way, which works in a similar way, but may have more issues with the lack of balance in this case.