Examples : I have a sentence in job description : "Java senior engineer in UK ".

I want to use a deep learning model to predict it as 2 categories : English and IT jobs. If i use traditional classification model, it only can predict 1 label with softmax function at last layer . Thus, i can use 2 model neural networks to predict "Yes"/"No" with both categories, but if we have more categories, it is too expensive . So do we have any deeplearning or machine learning model to predict 2 or more categories at same time ?

"Edit" : With 3 labels by traditional approach , it will be encoded by [1,0,0] but in my case, it will be encoded by [1,1,0] or [1,1,1]

Example : if we have 3 labels, and a sentences may be fit with all of these labels. So if output from softmax function is [0.45 , 0.35 , 0.2 ] we should classify it into 3 labels or 2 labels , or may be one ? the main problem when we do it is : what is good threshold to classify into 1, or 2 , or 3 labels ?

  • $\begingroup$ We have to use sigmoid function instead of softmax function. It can assign multiple classes to the data points. $\endgroup$ – NITISH MAHAJAN Mar 28 at 4:53

You can achieve this multi-label classification by replacing the softmax with a sigmoid activation and using binary crossentropy instead of categorical crossentropy as the loss function. Then you just need one network with as many output units/neurons as you have labels.

You need to change the loss to binary crossentropy as the categorical cross entropy only gets the loss from the prediction for the positive targets. To understand this, look at the formula for the categorical crossentropy loss for one example $i$ (class indices are $j$):

$ L_i = - \sum_j{t_{i,j} \log(p_{i,j})}$

In the normal multiclass setting, you use a softmax, so that the prediction for the correct class is directly dependent on the predictions for the other classes. If you replace the softmax by sigmoid this is no longer true, so negative examples (where $t_{i,j}=0$) are no longer used in the training! That's why you need to change to binary crossentropy, which uses both positive and negative examples: $L_i=-\sum_j{t_{i,j} \log(p_{i,j})} -\sum_j{(1 - t_{i,j}) \log(1 - p_{i,j})} $

| cite | improve this answer | |
  • $\begingroup$ why we need to using binary crossentropy instead of categorical crossentropy as the loss function ? can you explain more ? Now i am using sigmoid activation @robintibor $\endgroup$ – voxter Apr 11 '17 at 3:12
  • $\begingroup$ I have added an explanation to the answer @voxter $\endgroup$ – robintibor Apr 11 '17 at 11:57
  • $\begingroup$ Brilliant ! Thank you. Also, can you give some documents or tutorials which explain more math about functions in deeplearning like as you explained me ? $\endgroup$ – voxter Apr 12 '17 at 0:38
  • 1
    $\begingroup$ Great. These tutorials might help: neuralnetworksanddeeplearning.com deeplearning.net/tutorial deeplearning.stanford.edu/tutorial $\endgroup$ – robintibor Apr 12 '17 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.