Multiclass Classification using Binary Representation and Sigmoid Activation in Neural Network

Question

I am currently working on a multiclass classification problem where I have categorical variables that I've encoded using binary representations as follows:

0 -> 00
1 -> 01
2 -> 10
3 -> 11

My approach involves using a neural network with two output neurons, each having a sigmoid activation function. I use Mean Squared Error (MSE) as the loss function during training.

At inference time, I assign 0 as the output when the activation is less than 0.5; otherwise, I assign 1 for both neurons.

I am wondering if this approach is appropriate for a multiclass classification problem. The only advantage I can see now is fewer weights between the last and second last layer.

Answer 1

This could be clever. After all, more parameters means more potential to overfit, so if the parameter count can be reined in without sacrificing flexibility, this should be good news.

I see a few issues that are worth considering.

There is no natural interpretation of the explicit model outputs. You cannot interpret each neuron value as the probability of a particular outcome, and such probability values can be useful.

Consequently, you must rely on threshold-based classifications where being on the “wrong” side of a threshold results in a huge penalty. Similarly, there is no additional penalty for being extremely wrong as opposed to just slightly wrong.

If you go beyond four categories, you wind up with “empty” classifications. For instance, if you have five categories and do something with three binary neurons, you will wind up with three possibilities that do not correspond to any particular category. Sure, you can just regard those as mistakes when you are training and tuning the model, but it is not clear how to regard those when they are predicted by a deployed model.

You’re minimizing square loss instead of using the multinomial (log) likelihood as the optimization criterion. We’re in a lucky situation where we know form of the conditional distribution: multinomial on one roll of the die. This means that we can be confident about using maximum likelihood estimation and getting all of the benefits of maximum likelihood estimation. Frank Harrell has gone so far as to describe fitting a model like this by minimizing square loss as “silly”. Thus, while using such a model might be able to slightly reduce the parameter count, if those parameters are not estimated as well as they otherwise would have been, model performance might not improve as much as desired, if at all.