4
$\begingroup$

I'm working on a machine learning problem where I have to predict a categorical variable that has 12 different values, lets say A, B, C and so on.. (one thing I should mention is that one of the categories contain around 85% of the data)

I implemented two different aproaches but I'd like to ask you opinion on which one is better so I can focus on that particular one (Im using the XGBClassifier algorithm in python)

1) Keep th variable in one column, treating is as categorical as it is and build a model that predicts A, B, C, and so on... or

2) Create 12 columns on my dataset called "isA", "isB", "isC" and so on, where 11 of them will have value 0 and one of them will have value 1 depending on the category; them fit 12 models to my data that output the probability of belonging to that category (option 'binary:logistic')

one thing I realized is that if I ask the first model to predcit the probability of a category, and sum the 12 probabilities, it will sum up to 1, as in the second model that does not necessarily happens.

what do you guys think? whats the advantages and disadvantages of both aproaches?

$\endgroup$

2 Answers 2

2
$\begingroup$

Of course you should use the 1st approach. The classical way to do it is multinomial logistic regression (note that it is a synonym of softmax regression see : http://www.kdnuggets.com/2016/07/softmax-regression-related-logistic-regression.html).

The 2nd approach would be several binary logistic regressions. As you said the sum would not be 1. This is not recommended. As far as I know, nobody does it.

Note : there are other forms of multinomial regression other than logistic, but logistic is perfect for a start.

$\endgroup$
1
  • $\begingroup$ "As far as I know, nobody does it". - Almost every multiclass extensions of SVMs do that, you only need a robust probability coupling algorithm. Also, in the first approach there's no guarantee they would sum to 1 as well. $\endgroup$
    – Firebug
    Commented Sep 4, 2017 at 21:42
1
$\begingroup$

Go with the 2nd approach, which will generally produce better results. You want to read up on softmax. https://en.wikipedia.org/wiki/Softmax_function

The softmax function is widely used in Neuronetworks, to squash the sum of probabilities to 1.

A good theoretical read: https://chrisjmccormick.wordpress.com/2014/06/13/deep-learning-tutorial-softmax-regression/

$\endgroup$
2
  • 2
    $\begingroup$ Could you explain how or why the 2nd approach produces better results, or say something about "the advantages and disadvantages of both approaches"? $\endgroup$
    – rolando2
    Commented Jan 8, 2016 at 0:34
  • $\begingroup$ but on the second approach, the probabilities will not sum to 1... also I didn't understand if the second approach is called soft max or if you are suggesting a third one... As @rolando2, I'd like to know the advantages and disadvantages of both approaches. Thanks $\endgroup$
    – Italo
    Commented Jan 8, 2016 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.