1
$\begingroup$

I am working on some TV series data, so the number of records is very limited. I have 58 instances, one for each existing episode, which I have randomly split in 45 and 13. The main goal is to make a classification over five classes.

In order to make everything even worse, the data was unbalanced:

Feature    Dim
---------+-----
Class 1  | 24
Class 2  | 15
Class 3  |  7
Class 4  |  7
Class 5  |  5

This is why I applied SMOTE to the training data, but this keeps me with the problem of having a very small testing dataset. In fact, this is how the test data is distributed:

Feature    Dim
---------+-----
Class 1  |  5
Class 2  |  3
Class 3  |  2
Class 4  |  2
Class 5  |  1

How can I measure performance on such little data? I can't tell how many "Class 5" elements it is able to recognize, because there is just one element.

Is there a way to handle this?

$\endgroup$
6
  • 1
    $\begingroup$ This will require a large amount of background reading as you have violated a whole variety of statistical principles. Sorry to not have good news. Check out RMS and fharrell.com/post/classification and note that the minimum sample size for data splitting to work well as a validation method is on the order of n=20,000. You're too subject to "the luck of the split" and are using methods that are harmed by imbalance. Good methods are not hurt by imbalance except for having a lower effective sample size that makes standard errors larger. $\endgroup$ Jun 21, 2022 at 23:15
  • $\begingroup$ Do you have data about one single TV series? In other words, what is the population of TV series that you are studying? $\endgroup$
    – dipetkov
    Jun 22, 2022 at 23:45
  • $\begingroup$ @dipetkov Thank you for your interest. I have the data about each existing episode of a tv series, 58 episodes $\endgroup$
    – Jonathan
    Jun 22, 2022 at 23:48
  • $\begingroup$ I'm not sure I understand what problem you are trying to solve. Is this series going to have more seasons? If the series is concluded, then why are you building a classification model? $\endgroup$
    – dipetkov
    Jun 22, 2022 at 23:53
  • $\begingroup$ It's not really anything that will be used, it is to demonstrate that it could be possible to forecast the popularity of an episode by using some features extracted from scripts/video $\endgroup$
    – Jonathan
    Jun 22, 2022 at 23:55

1 Answer 1

3
$\begingroup$

Train/test split sounds like a bad idea indeed. I'd try jackknife resampling/LeaveOneOut cross-validation for this case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.