# Overfitting on a balanced dataset

I have a balanced dataset containing 12 different classes (~11000 entries with 104 features). I use PCA to reduce the feature space to 20 features and I am shuffling the feature matrix and the labels to create a training (70% of the data), a test (20% of the data) and a evaluation (10% of the data) sets in order to test the performance of my classifier. I use a SVM with radial basis kernel and the one-vs-one strategy.

Each of these entries represent an audio snippet. The goal is to find snippets of audio that belong to any of these classes when a new audio segment is given. Class 0 represent the "unknown" class. The unseen audio segments contain snippets from these classes in known locations and I want to be able to tell when these events occur and the class they belong to.

I use sklearn for all the computations and algorithms. When the training is complete I use the test and I get the following result (classification_report from sklearn, with one row per class)

         precision    recall  f1-score   support

0.0       0.98      0.49      0.65       352
1.0       0.66      0.82      0.73       148
2.0       0.63      0.80      0.70       149
3.0       0.77      0.66      0.71       235
4.0       0.60      0.81      0.69       145
5.0       0.69      0.53      0.60       249
6.0       0.75      0.84      0.79       168
7.0       0.53      0.66      0.59       150
8.0       0.58      0.76      0.66       148
9.0       0.58      0.62      0.60       181
10.0       0.99      1.00      0.99       177
11.0       0.76      0.87      0.81       166

avg / total       0.74      0.71      0.70      2268


Let's assume that this result is acceptable for now.

When I use unseen data to classify each snippet, the classifier always outputs label 6 as an output. This shouldn't be the case as the unseen audio segment contains snippets that belong to more than one class.

1. Am I right to assume that the model is overfitting?
2. Given that this is a balanced dataset the model is not biased towards a specific class. How come the result is always fixed to one label?
3. Is my way of dividing the data wrong? If so, why is that?

Thanks

• How do you use the one-vs-all predictions in order to predict the concept? It looks like the problem is over there. Do you use the classes ration in the population (e.g., as in Naive Bayes)? – DaL Mar 15 '17 at 8:29
• During the training phase each audio segment is broken down into smaller audio snippets and features are extracted from it and each snippet gets a class label. During the testing phase each audio snippet of the unseen audio segment should be assigned a label after the classification takes place. Since I know the locations of the audio snippets and the class they belong too I want to use these metrics to determine how well by system is performing. – BitWhyz Mar 15 '17 at 9:14
• Before we move to the performance metrics, I still don't understand how do you make the prediction. say that you got a new snippet and it is a hit a of 3 classes (or you get a confidence level from all classifiers). What will be the prediction? What will be the aggregation logic? – DaL Mar 15 '17 at 13:09
• I apologize for my mistake. I meant to say classification and not prediction. One snippet can belong to only 1 class. Segments can have different snippets that belong to various classes. The model tries to assign a label to each snippet. – BitWhyz Mar 15 '17 at 13:41
• You wrote "I use a SVM with radial basis kernel and the one-vs-one strategy" (by the way, one-vs-all might be more useful here). So you have plenty of classifications for class 1 vs class 7, etc. How do you transform all these classifications into a single one? It seems that your problem in around that area. – DaL Mar 15 '17 at 14:27

It's been a while before my last post. I think I have figured out what was causing the problem. Class 0 in my dataset was a negative class and it was confusing the training process. As soon as I removed it the predictions made more sense.

Looking at the image below you can see how one feature is distributed in all 11 classes. Class 0 used to look like a single Gaussian distribution spanning across the whole x axis.

I have used a Naive Bayes model as well and the results confirmed my previous findings.

When I use unseen data to make a new prediction, the classifier always outputs label 6 as a prediction. Obviously that is not acceptable.

This isn't obvious. Perhaps it really is most accurate to ignore the features and just predict 6 every time. This is one reason that when evaluating predictive accuracy, you should compare your substantive models to a trivial model.

1. No, I don't see how you would have come to that conclusion from what you've presented here.

2. I can't tell from the sparse information you've presented here. You might try adding details. You might also try using a simpler model such as logistic regression, seeing if it also constantly predicts 6, and looking at its coefficients.

3. Your way of dividing the data sounds fine.

• @BitWhyz While the information you added is helpful, I still don't see why you would assume overfitting has occurred, nor could I tell you why your model is constantly predicting 6. Try using a trivial model, and a simpler but nontrivial model with coefficients you can look at. – Kodiologist Mar 15 '17 at 14:29
• my rational is that since I have trained a model on a balanced set of 12 classes it should, to some extend, be able to do that. The precision and recall for most of the classes is above 50% which made me think that the model is able to classify an audio snippet to the correct class (with some error of course). Is this assumption irrational? I am testing a simpler model as you have suggested. – BitWhyz Mar 16 '17 at 7:43
• @BitWhyz I'm not sure—to judge overall performance, I'd need something like the percentage of all cases classified correctly. (Overfitting, of course, is when the overall training performance improves at the expense of overall test performance.) You might find this helpful, although your case is different because you have equal base rates of the 12 classes: stats.stackexchange.com/questions/213017 – Kodiologist Mar 16 '17 at 14:31
• isn't the overall performance of this model given by the f-score as I showed in the matrix? Or do you mean that it's not conclusive? – BitWhyz Mar 16 '17 at 15:00
• @BitWhyz Since you're doing classification, you presumably want the proportion classified correctly. You should compute it on your test set, too. – Kodiologist Mar 16 '17 at 15:48