0
$\begingroup$

My background is in ecology, it is common to have smaller sample sizes and class imbalances and ML approaches are still increasingly adopted. My specific dataset: training set is 49 sample, my test set is 16 samples - total 65 samples. I have also used loocv (not reported in the table) where I observed similar pattern.

My class proportions are as such across 6 classes, when I stratify it 75-25%, it is to make sure class 5 (rare class) is represented in training-test split and stratified k-folds as well for loocv and 5-fold cv:

class 1: 19 samples

class 2: 9 samples

class 3: 9 samples

class 4: 15 samples

class 5: 5 samples

class 6: 8 samples

I know accuracy is low in general - I have attempted to optimize it better using different variable selection methods, etc. I understand its not suitable for ML in general but technically, I was curious what the issue could be - just the size of samples, should stratification be done different for training-test split, etc.

All analysis in R programming environment.

Thank you in advance.

Accuracy metrics

$\endgroup$
0
$\begingroup$

I am almost certain this is just pure luck. With only 16 samples, there is very little stability in results. Another way to think about it is that the improvement from train to test is not statistically significant (probably).

I would be curious to see the CV results, where you would at least have 65 samples. With such a small data set, I would a) stick with simpler, regularized models (don't do ANNs), and b) inject as much subject matter expertise as you can. Maybe you can just pick features based on your intuition on which ones should be important, since you don't have the luxury of extra validation data to do variable selection (and even if you did, differences in the results of different feature sets could just random noise).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.