# Which neural network is better?

This is a homework question where I have not been able to reach any conclusion. I have an exam tomorrow. Please help me out.

We have a set of data from patients who have visited a hospital. A set of features (e.g., temperature , height...) have been also extracted for each patient. Our goal is to decide whether a new visiting patient some $n$ number of diseases or not.

This problem is to be solved using neural network. We have two choices: either to train a separate neural network for each of the diseases or to train a single neural network with one output neuron for each disease, but with a shared hidden layer. Which method do you prefer? Justify your answer .

My opinion: Both methods are almost the same because we have to train the same number of weights in both case. But in one case we can train them all at once and in another we will train them individually so for training it will take more time.

Hint: The main difference between using one network for all the diseases is that in that case, the hidden layer might learn some features that can be shared between the disease-predicting outputs.

Why do you think training individual models will take more time?

• Agreed. As an addendum I'd like to ask in the case where the goal is a medical application, which do you think is most important - training time, or accuracy of results?
– Pat
Feb 20 '14 at 13:28
• Accuracy of results matter more here Feb 20 '14 at 13:30
• I agree. So that's something your answer needs to explicitly deal with. Another factor, which I suspect isn't the thrust of the question but I personally would mention in passing, is which method is better at dealing with the client changing requirements. In practical applications this kind of thing happens a lot. What if in a year's time the hospital wants to detect a new disease using the same input data, for example? Which method can most easily be amended to accommodate this?
– Pat
Feb 20 '14 at 13:58

If the diseases are mutually exclusive (i.e. the probability of having more than one disease at the same time is negligible), e.g. for differential diagnosis, then using a single network with a softmax activation function in the output layer would be a good idea. That would make the shared hidden units more effective, if some combination of hidden units suggested that disease A was likely, that would force the outputs for the other diseases closer to zero.

Personally I try not to worry about training time (provided it remains feasible) as it is better to get a good answer slowly than a bad answer quickly. This is especially true in a medical context where errors may have severe consequences.

Depends on how your data's distributed and if your outcomes are correlated (or not).

Why not try out both methods and convince yourself which one is better and why. There's no free lunch and sometimes the results can be completely different from your intuitions / hypothesis.

Here are some general hints:

Make sure you get your experiment methodology right and split your data into training+validation / testing or cross-validation / testing. Log everything: all (hyper)parameters and loss evolution (convergence) for all experiments. Trial and error. Cause and effect. So change only one thing at a time.

You might also want to look into things like dropout and observe & test it's effects (look into ensembles) on the network. The independent networks need not be fully independent.

Finally:

Here are some questions you might want to ask yourself after all the effort: is there any relationship between the various outcomes (diseases) or not? If not, is there any point to have shared parameters with other NN models? What is the optimal architecture then? If this were a real world setting how could I make sure that I could add new diseases easily to the system? Perhaps there might be groups of diseases that correlate well? How could I find these groups by using a single network? What are the features that correlate with these groups?

This is probably more than you need for the homework, the point I am trying to make is that you need to have some hypothesis, test it, then repeat. Nobody can tell you anything for sure. Until you test it, it's all speculation.

Good luck and enjoy the process. I hope my answer will motivate you to try and find out more by yourself.