How to approach a classification problem when the sample size is only about 50

Question

I was given a data consists of 53 people and I was asked to come up with a general classification rule based on biomarkers that can be used to classify each person under one of the three possible causes for their illness.

My questions are:

1. I was thinking of making a use of classification tree via random forest, but I realized that the size of my data is only about 50, and that this can cause overfitting on the data in hand. What are the classification methods that can be taken on the data that is consisted only of ~50 observations?

2. Instead of trying to do the classification, would something like multinomial regression (in this case, response will be nominal, 1 = 1st type of cause for illness, 2= 2nd type of cause for illness, 3= 3rd type of cause for illness) make more sense for this sample size?

3. Should I still be dividing my original data into a test and a training set? Like I mentioned, I think the sample size is too small for this.

Thank you,

Answer 1

This is not a good situation for classification as detailed here. Biomarker research is all about tendencies, not about forced choices. And note that just to estimate an overall tendency, e.g., a simple unconditional probability, requires a minimum of 96 subjects as detailed in my RMS course notes and in BBR which also has a chapter on complexities of analyzing high dimensional data, especially about the stability of features "selected".

Think of 96 as the minimum number of subjects needed to estimate just the intercept in a binary logistic model. And that only achieves a margin of error (with 0.95 confidence) of $\pm 0.1$ in the predicted risk of outcome/disease. To nail down the outcome probability to within a margin of error of $\pm 0.05$ requires $4\times$ that many subjects.

The above is for the 2-outcome case. For 3 outcomes things are more challenging. If the outcomes are not ordered you would be right to use polytomous (multinomial) logistic regression.

Yes the dataset is too small by a factor of perhaps 500 for data splitting to be reliable. You might consider 100 repeats of 10-fold cross-validation, or 500 bootstrap iterations. For either of these resampling techniques all supervised learning steps must be repeated afresh for each iteration.

As a rough guess, the total sample size needed for a 3-class unordered problem such that the smallest outcome category has at least $15 \times$ the number of candidate features.