What is the difference between using single multi-output NN and multiple single-output NNs

Question

Pretty new to machine learning and would like to know what is the difference in model accuracy between using single multi-output NN and multiple single-output NNs all used in tandem (OvA and OvO)?

Eg. Say there is a problem where there is a set of input medical diagnosis codes

Xd with sample vectors like [dx1, ..., dx5]

and a set of "correct" medical diagnosis codes that the the input codes commonly need to be changed to

Yd with sample vectors also like [dy1, ..., dy5]

and I'd like to train a NN to predict all of the correct diagnoses.

The way I see it, there are two options, a single NN with multiple outputs each trained on one of the different elements of the answer vectors yd OR using multiple NN models all used in tandem each with one output trained on one of the different elements of the answer vectors yd.

In my inexperienced view, the single multi-output NN would be better since the inputs and back-propagations have more interaction with each other, but I really don't know. Could somebody help me out and explain the differences, if any? Thanks.

Answer 1

The multiple-out NN probably will have lower model capacity per output than the single-out NNs will. However, adding more capacity is usually pretty easy.

Especially if there is limited data, the multiple-out NN may work better because of knowledge transfer. The NN may learn hidden representations which are useful for predicting all of the outputs of the NN. Therefore, these representations (in the hidden layer(s)) contain knowledge that is trained and used by all of the data.

Intuitively, hand-eye coordination is important for both pouring water into a cup and turning the door knob. If there were two brains in my head, one for pouring water and one for turning the door, I would probably have to learn hand-eye coordination twice. This is like the single-NNs case. On the other hand, with a single multioutput NN, like a brain which can do many tasks, I only need to learn this skill once, and I may learn it better, since I will be using the skill for both tasks instead of just one.