1
$\begingroup$

I used SVM to predict the ranking score of muffin recipes. X is a numpy array of ingredient amounts of a certain recipe and y is the label according to the online ranking score. First I labelled my data in two classes like this:

ranking < 3.5 - label = 0
ranking > 3.5 - label = 1

Then I labelled my data like this:

ranking < 3.5 - label = 0 
ranking between 3.5 & 4.25 - label = 1
ranking > 4.25 - label = 2

By doing this the accuarcy decreased by 20%! How is this possible? Dividing my data in more classes should have led to a higher accuracy score right? How can we explain this?

$\endgroup$
  • 2
    $\begingroup$ Why did you recode the data at all? Why not simply use some kind of regression? Binning the data is generally not a recommended approach. $\endgroup$ – Tim Jul 8 '16 at 11:30
  • 1
    $\begingroup$ Try, at least, SVM and Random Forest regression. $\endgroup$ – Firebug Jul 8 '16 at 11:57
3
$\begingroup$

Usually, classification will become more difficult when increasing the amount of classes for the same samples and features - because this means there are more options for the target variable (= more possibilities for confusion). In your case: in the first scenario, there was no confusion possible within the 1 class, but in the second scenario, there is, because those belong to different classes now. Therefore, if you count confusion of 1 and 2 as error that you measure within the one, overall accuracy, you definitely made the problem harder - as you now have more possibilities to make errors using the same samples and features (simplest example: have 1 constant class label - no error possible at all).

BTW: with more than 2 classes you might need to look at other metrics than one scalar performance value that gives you more information about what is going on anyway. Consider using a confusion matrix (and poss. looking at the distribution at TPR/TNR, ROC/AUC etc. over different classes).

$\endgroup$
  • $\begingroup$ Thank you geekoverdose! This makes everything more clear! I am going to use the learning curve and a graphical representation of a confusion matrix to find out what is going on. Do you think I can conclude my data is too complex to be classified in more classes and that the ranking score can therefore be not predicted very accurate to the actual ranking? $\endgroup$ – Afke Jul 8 '16 at 11:01
  • 1
    $\begingroup$ This depends on what your goal is: it needs to be reflected in the target variable, and the prediction performance should be sufficient for the goal as well - you define what to achieve in your goal. Usually you will only do such conclusions after a thorough evaluation (e.g. using different scenarios with different target variables, tested on different model types and hyperparameters with e.g. repeated cross validation). Only from seeing how prediction performance behaves over those different options I would conclude something - otherwise it might be based on too less information :) $\endgroup$ – geekoverdose Jul 8 '16 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.