# Softmax Loss vs Binary Loss for classification?

I was trying to understand the final section of the paper "Revisiting Baselines for Visual Question Answering". Authors state that their model performs better with a binary loss in comparison to a softmax loss. I wonder what actually a binary loss mean in this case? I think the softmax loss is the same term for binary cross-entropy. Could someone explain what exactly a binary loss is? Thanks.

• Welcome to the site. This is two questions, one of which (about tensorflow) is off topic. If you remove that part, it will be a fine question. – Peter Flom Jan 16 at 10:55
• @PeterFlom Done! – hexpheus Jan 16 at 18:36

There is a nice explanation here

Binary Cross-Entropy Loss is also called Sigmoid Cross-Entropy loss. It is a Sigmoid activation plus a Cross-Entropy loss. Unlike Softmax loss it is independent for each vector component (class), meaning that the loss computed for every vector component is not affected by other component values.

The term binary stands for number of classes = 2.

I think that binary loss is the one based Shannon on entropy $$-\sum_ip_i\ln p_i$$, while the softmax is based on Boltzmann distribution: $$\frac{e^{z_i}}{\sum_j e^{z_i}}$$
Softmax itself shouldn't be a loss function though. It's the probability like function that you use to pick the output of classificator. For instance, you could use it to calculate probabilities $$\hat p_{ij}$$ of classificator outcomes for categories $$i$$ for sample $$j$$.
Then you can use the entropy based loss function to evaluate the fit: $$\sum_ip_{ij}\ln\hat p_{ij}$$, where $$p_{ij}$$ is a binary outcome of a category $$i$$.