I have a balanced dataset containing 12 different classes (~11000 entries with 104 features). I use PCA to reduce the feature space to 20 features and I am shuffling the feature matrix and the labels to create a training (70% of the data), a test (20% of the data) and a evaluation (10% of the data) sets in order to test the performance of my classifier. I use a SVM with radial basis kernel and the one-vs-one strategy.
Each of these entries represent an audio snippet. The goal is to find snippets of audio that belong to any of these classes when a new audio segment is given. Class 0 represent the "unknown" class. The unseen audio segments contain snippets from these classes in known locations and I want to be able to tell when these events occur and the class they belong to.
I use sklearn for all the computations and algorithms. When the training is complete I use the test and I get the following result (classification_report from sklearn, with one row per class)
precision recall f1-score support
0.0 0.98 0.49 0.65 352
1.0 0.66 0.82 0.73 148
2.0 0.63 0.80 0.70 149
3.0 0.77 0.66 0.71 235
4.0 0.60 0.81 0.69 145
5.0 0.69 0.53 0.60 249
6.0 0.75 0.84 0.79 168
7.0 0.53 0.66 0.59 150
8.0 0.58 0.76 0.66 148
9.0 0.58 0.62 0.60 181
10.0 0.99 1.00 0.99 177
11.0 0.76 0.87 0.81 166
avg / total 0.74 0.71 0.70 2268
Let's assume that this result is acceptable for now.
When I use unseen data to classify each snippet, the classifier always outputs label 6 as an output. This shouldn't be the case as the unseen audio segment contains snippets that belong to more than one class.
- Am I right to assume that the model is overfitting?
- Given that this is a balanced dataset the model is not biased towards a specific class. How come the result is always fixed to one label?
- Is my way of dividing the data wrong? If so, why is that?
Thanks