0
$\begingroup$

I am doing a classification task in Python to classify audio files of different musical instrument into their respective class, in my case there are 4 class, which are Brass, String, Percussion, and Woodwind. I used SVM algorithm as the classifier. My code looks a bit like this (I do not change any parameter for the classifier):

#X is feature matrix, y is class vector
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

#SVM Classifier
svm = SVC()
svm.fit(X_train,y_train)
svm_pred = svm.predict(X_test)
print(metrics.classification_report(y_test,svm_pred)

When I try to run this code, I got problem with the classifier. The error code looks like this:

            precision  recall   f1-score   support

Brass         1.00      0.21      0.34        72
Percussion    0.38      1.00      0.55       279
String        1.00      0.15      0.26       276
Woodwind      0.00      0.00      0.00       156

avg / total   0.58      0.43      0.32       783

C:\Users\Anaconda3\lib\site-packages\sklearn\metrics\classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.

When I checked my predicted labels from SVM classifier(svm_pred), no Woodwind class are predicted by the classifier

>>> set(svm_pred)
{'Brass','String','Percussion'}

My number of samples for each class are like this: Brass = 200 samples, Woodwind = 500 samples, Percussion = 900 samples, and String = 800 samples so it is a bit imbalanced

My question is, is it possible for a SVM classifier to not predict a class at all in the output of the classifier like my case above?

$\endgroup$
  • $\begingroup$ Can you try scaling the inputs? I find SVMs do better when features are scaled. $\endgroup$ – Demetri Pananos May 4 at 3:02
1
$\begingroup$

First, you write "I do not change any parameter for the classifier." This is poor practice. Different problems have different optimal hyperparameter configurations. This is because we need to find the optimal for your particular data.

Second, you need a method to relate the inputs (audio signals) to the model that you're using. The absolutely generic application of an SVM is comparing vectors. Are the audio files you're using represented as single vectors, or are they represented in some more complex format (perhaps a matrix)? Also, the default SVM kernel in sklearn is the RBF kernel. Does this kernel make sense for your inputs, or would an alternative kernel return more relevant comparisons of your inputs? Could a pre-processing step help extract more useful signal? These are all research questions you'll have to consider as you pursue your project.

Third, the predict method uses a default number to decide at what level of predicted outcome the sample should be allocated to a class. What this means is that the classification is sensitive to the choice of cutoff, and choosing a different cutoff will yield a different classification.

In other words, it's worth considering what the consequences are for a correct or incorrect decision, and selecting classification rules which minimize your risk. This is true regardless of whether or not you think the process of audio reproduction is deterministic, because some problems may have steep costs for misclassifying specific instruments, but not others.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I would argue that this is actually a good case for binary decisions. Frank Harrell writes in the linked blog post "When are forced choices appropriate? I think that one needs to consider whether the problem is mechanistic or stochastic/probabilistic... It may be best to apply classification techniques instead just to high signal:noise ratio situations such as those in which there there is a known gold standard and one can replicate the experiment and get almost the same result each time.". Classifying instrument sounds is probably one of those cases. $\endgroup$ – Demetri Pananos May 4 at 3:01
  • $\begingroup$ @DemetriPananos What is special about musical instruments that makes this a good case for binary decisions? $\endgroup$ – Sycorax May 4 at 3:05
  • $\begingroup$ Instrument sounds are not probabilistic; they are mechanistic. When you play a saxophone, its going to sound similar to most all other saxophones you've heard (save any pathological examples). Its much like image recognition. Again, referencing Frank's post "An example [of when binary decisions my be appropriate] is pattern recognition - visual, sound, chemical composition, etc". If you are arguing otherwise, it seems your first link undermines your point. $\endgroup$ – Demetri Pananos May 4 at 3:08
  • $\begingroup$ What about if the features you have are only weakly informative about the outcome? If our assumption is that all saxophones sound alike, and all woodwinds sound alike, it seems like we would have a higher-quality model. But in this case, there’s a lot of confusion about what woodwinds sound like. In this setting, a probabilistic interpretation seems more appropriate because the features are so weak. $\endgroup$ – Sycorax May 4 at 3:10
  • $\begingroup$ I'm fairly confident that audio files of instruments are not weakly informative about the source of the sound. But, until OP posts data, that remains to be seen. $\endgroup$ – Demetri Pananos May 4 at 3:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.