I am currently working on a multiclass classification problem where I have categorical variables that I've encoded using binary representations as follows:

0 -> 00
1 -> 01
2 -> 10
3 -> 11

My approach involves using a neural network with two output neurons, each having a sigmoid activation function. I use Mean Squared Error (MSE) as the loss function during training.

At inference time, I assign 0 as the output when the activation is less than 0.5; otherwise, I assign 1 for both neurons.

I am wondering if this approach is appropriate for a multiclass classification problem. The only advantage I can see now is fewer weights between the last and second last layer.

  • $\begingroup$ how many classes do you have? $\endgroup$
    – gunes
    Jul 22, 2023 at 12:55
  • $\begingroup$ Assume 2^k, k <- N. $\endgroup$ Jul 22, 2023 at 12:58
  • $\begingroup$ why do you have two output neurons if you have 2^k classes? $\endgroup$
    – gunes
    Jul 22, 2023 at 13:57
  • $\begingroup$ assume k output neurons for 2^k classes $\endgroup$ Jul 22, 2023 at 14:10
  • $\begingroup$ Does one neuron consider if the category is dog vs cat and the other neuron consider if the photo is daytime or nighttime (so to speak)? Or do you have four mutually exclusive categories? If you are in the latter case, what would you do if you had a number of categories that isn’t equal to an integer power of two? $\endgroup$
    – Dave
    Jul 22, 2023 at 15:21

1 Answer 1


This could be clever. After all, more parameters means more potential to overfit, so if the parameter count can be reined in without sacrificing flexibility, this should be good news.

I see a few issues that are worth considering.

There is no natural interpretation of the explicit model outputs. You cannot interpret each neuron value as the probability of a particular outcome, and such probability values can be useful.

Consequently, you must rely on threshold-based classifications where being on the “wrong” side of a threshold results in a huge penalty. Similarly, there is no additional penalty for being extremely wrong as opposed to just slightly wrong.

If you go beyond four categories, you wind up with “empty” classifications. For instance, if you have five categories and do something with three binary neurons, you will wind up with three possibilities that do not correspond to any particular category. Sure, you can just regard those as mistakes when you are training and tuning the model, but it is not clear how to regard those when they are predicted by a deployed model.

You’re minimizing square loss instead of using the multinomial (log) likelihood as the optimization criterion. We’re in a lucky situation where we know form of the conditional distribution: multinomial on one roll of the die. This means that we can be confident about using maximum likelihood estimation and getting all of the benefits of maximum likelihood estimation. Frank Harrell has gone so far as to describe fitting a model like this by minimizing square loss as “silly”. Thus, while using such a model might be able to slightly reduce the parameter count, if those parameters are not estimated as well as they otherwise would have been, model performance might not improve as much as desired, if at all.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.