0
$\begingroup$

I have a balanced dataset containing 12 different classes (~11000 entries with 104 features). I use PCA to reduce the feature space to 20 features and I am shuffling the feature matrix and the labels to create a training (70% of the data), a test (20% of the data) and a evaluation (10% of the data) sets in order to test the performance of my classifier. I use a SVM with radial basis kernel and the one-vs-one strategy.

Each of these entries represent an audio snippet. The goal is to find snippets of audio that belong to any of these classes when a new audio segment is given. Class 0 represent the "unknown" class. The unseen audio segments contain snippets from these classes in known locations and I want to be able to tell when these events occur and the class they belong to.

I use sklearn for all the computations and algorithms. When the training is complete I use the test and I get the following result (classification_report from sklearn, with one row per class)

         precision    recall  f1-score   support

    0.0       0.98      0.49      0.65       352
    1.0       0.66      0.82      0.73       148
    2.0       0.63      0.80      0.70       149
    3.0       0.77      0.66      0.71       235
    4.0       0.60      0.81      0.69       145
    5.0       0.69      0.53      0.60       249
    6.0       0.75      0.84      0.79       168
    7.0       0.53      0.66      0.59       150
    8.0       0.58      0.76      0.66       148
    9.0       0.58      0.62      0.60       181
   10.0       0.99      1.00      0.99       177
   11.0       0.76      0.87      0.81       166

avg / total       0.74      0.71      0.70      2268

Let's assume that this result is acceptable for now.

When I use unseen data to classify each snippet, the classifier always outputs label 6 as an output. This shouldn't be the case as the unseen audio segment contains snippets that belong to more than one class.

  1. Am I right to assume that the model is overfitting?
  2. Given that this is a balanced dataset the model is not biased towards a specific class. How come the result is always fixed to one label?
  3. Is my way of dividing the data wrong? If so, why is that?

Thanks

Improve this question
$\endgroup$
7
  • $\begingroup$ How do you use the one-vs-all predictions in order to predict the concept? It looks like the problem is over there. Do you use the classes ration in the population (e.g., as in Naive Bayes)? $\endgroup$
    – DaL
    Mar 15, 2017 at 8:29
  • $\begingroup$ During the training phase each audio segment is broken down into smaller audio snippets and features are extracted from it and each snippet gets a class label. During the testing phase each audio snippet of the unseen audio segment should be assigned a label after the classification takes place. Since I know the locations of the audio snippets and the class they belong too I want to use these metrics to determine how well by system is performing. $\endgroup$
    – BitWhyz
    Mar 15, 2017 at 9:14
  • $\begingroup$ Before we move to the performance metrics, I still don't understand how do you make the prediction. say that you got a new snippet and it is a hit a of 3 classes (or you get a confidence level from all classifiers). What will be the prediction? What will be the aggregation logic? $\endgroup$
    – DaL
    Mar 15, 2017 at 13:09
  • $\begingroup$ I apologize for my mistake. I meant to say classification and not prediction. One snippet can belong to only 1 class. Segments can have different snippets that belong to various classes. The model tries to assign a label to each snippet. $\endgroup$
    – BitWhyz
    Mar 15, 2017 at 13:41
  • $\begingroup$ You wrote "I use a SVM with radial basis kernel and the one-vs-one strategy" (by the way, one-vs-all might be more useful here). So you have plenty of classifications for class 1 vs class 7, etc. How do you transform all these classifications into a single one? It seems that your problem in around that area. $\endgroup$
    – DaL
    Mar 15, 2017 at 14:27

2 Answers 2

Reset to default
1
$\begingroup$

It's been a while before my last post. I think I have figured out what was causing the problem. Class 0 in my dataset was a negative class and it was confusing the training process. As soon as I removed it the predictions made more sense.

Looking at the image below you can see how one feature is distributed in all 11 classes. Class 0 used to look like a single Gaussian distribution spanning across the whole x axis.

I have used a Naive Bayes model as well and the results confirmed my previous findings.

Histogram of 1 feature for 12 classes

Improve this answer
$\endgroup$
0
$\begingroup$

When I use unseen data to make a new prediction, the classifier always outputs label 6 as a prediction. Obviously that is not acceptable.

This isn't obvious. Perhaps it really is most accurate to ignore the features and just predict 6 every time. This is one reason that when evaluating predictive accuracy, you should compare your substantive models to a trivial model.

  1. No, I don't see how you would have come to that conclusion from what you've presented here.

  2. I can't tell from the sparse information you've presented here. You might try adding details. You might also try using a simpler model such as logistic regression, seeing if it also constantly predicts 6, and looking at its coefficients.

  3. Your way of dividing the data sounds fine.

Improve this answer
$\endgroup$
5
  • $\begingroup$ @BitWhyz While the information you added is helpful, I still don't see why you would assume overfitting has occurred, nor could I tell you why your model is constantly predicting 6. Try using a trivial model, and a simpler but nontrivial model with coefficients you can look at. $\endgroup$
    – Kodiologist
    Mar 15, 2017 at 14:29
  • $\begingroup$ my rational is that since I have trained a model on a balanced set of 12 classes it should, to some extend, be able to do that. The precision and recall for most of the classes is above 50% which made me think that the model is able to classify an audio snippet to the correct class (with some error of course). Is this assumption irrational? I am testing a simpler model as you have suggested. $\endgroup$
    – BitWhyz
    Mar 16, 2017 at 7:43
  • $\begingroup$ @BitWhyz I'm not sure—to judge overall performance, I'd need something like the percentage of all cases classified correctly. (Overfitting, of course, is when the overall training performance improves at the expense of overall test performance.) You might find this helpful, although your case is different because you have equal base rates of the 12 classes: stats.stackexchange.com/questions/213017 $\endgroup$
    – Kodiologist
    Mar 16, 2017 at 14:31
  • $\begingroup$ isn't the overall performance of this model given by the f-score as I showed in the matrix? Or do you mean that it's not conclusive? $\endgroup$
    – BitWhyz
    Mar 16, 2017 at 15:00
  • $\begingroup$ @BitWhyz Since you're doing classification, you presumably want the proportion classified correctly. You should compute it on your test set, too. $\endgroup$
    – Kodiologist
    Mar 16, 2017 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.