I am working on some TV series data, so the number of records is very limited. I have 58 instances, one for each existing episode, which I have randomly split in 45 and 13. The main goal is to make a classification over five classes.
In order to make everything even worse, the data was unbalanced:
Feature Dim
---------+-----
Class 1 | 24
Class 2 | 15
Class 3 | 7
Class 4 | 7
Class 5 | 5
This is why I applied SMOTE to the training data, but this keeps me with the problem of having a very small testing dataset. In fact, this is how the test data is distributed:
Feature Dim
---------+-----
Class 1 | 5
Class 2 | 3
Class 3 | 2
Class 4 | 2
Class 5 | 1
How can I measure performance on such little data? I can't tell how many "Class 5" elements it is able to recognize, because there is just one element.
Is there a way to handle this?
