Confused in selecting the number of hidden layers and neurons in an MLP for a binary classification problem

Question

I'm working on a disease classification dataset which has 25 features including the class attribute. It is a binary classification problem. The dataset has total 300 training instances.

I trained a feed-forward neural network with the following architecture:

model = Sequential()

model.add(Dense(100, input_dim=train_x.shape[1], activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(25, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(train_x.values, train_y.values, epochs=50, batch_size=40, validation_split=.2, verbose=0)

The architecture has 3 hidden layers with 100, 50 and 25 hidden neurons in the layers.

After training, the model shows the following scores on the test set which includes 100 test instances.

100/100 [==============================] - 0s 94us/step
Test loss score: 0.02940008039586246
Test accuracy: 100.0

So, I've got 100% accuracy with this architecture.

The number of hidden layers and hidden neurons per layer was taken arbitrarily.

However, I have found something in an article of Jeff Heaton that:

and also found some rules about choosing the number of hidden neurons per layer:

1. The number of hidden neurons should be between the size of the input layer and the size of the output layer.

2. The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.

3. The number of hidden neurons should be less than twice the size of the input layer.

These three rules provide a starting point for you to consider.

Now my question is:

1. Is my model architecture acceptable?

2. According to Heaton, we can have more than 2 hidden layers for sort of automatic feature engineering. So is using 3 layers okay or not?

3. Here, I've used 100, 50 and 25 neurons in the hidden layers arbitrarily. The output layer contains only 1 neuron as it is a binary classification. But according to the thumb rule, the number of hidden neurons should be between the size of the input layer and the size of the output layer. So, according to this rule, the number of hidden neurons for this dataset should be between 1-24 as there are 24 input features and the 1 output feature. So, using 100, 50, 25 hidden neurons is wrong?

4. For this dataset of 300 instances with 24 features and a class attribute of a binary class, what should be the number of hidden neurons? Is there any other empirical rule that I can follow?

5. Using 100, 50 and 25 hidden neurons yields a good result. So should I decrease the number of neurons? Why?

Answer 1

In Deep Learning there are no hard & fast rules to set the number of layers, the number of hidden units per layer and not even the kind of connections between layers: who claims the contrary often doesn't have experience of struggling with modern architectures, and relies on insight 20 years (if not more) old. "Proofs" of my claim:

• you have all kinds of different architectures performing increasingly better on ImageNet: VGG, Inception, ResNet, ResNext, Xception, DenseNet, etc. Even the simplest empirical rules such as "double the number of channels before pooling" or even just "use pooling after a convolutional layer" aren't valid in general, though replacing pooling with increasing strides in successive layers has its drawbacks

• if the "back-of-the-envelope" rules to choose the number of hidden layers & neurons from the days of old were still valid today, we wouldn't have so much investment on automated architecture learning - I think after Deep RL and GANs, this is currently the single biggest R&D investment in AI companies.

But that's not necessarily bad news: your architecture is getting excellent test set accuracy as it is, so maybe you just don't have to worry about optimal architecture. Accuracy is not the correct metric to look at for classification, but as you have nearly balanced classes, and anyway astonishing classification accuracy, it's not like using another metric will make a big difference.

Since you haven't provided some data, or some training curves, it's hard to say if overfitting to the training set is actually happening. However, there are a few checks you could make, to clear any doubts. Since your architecture is relatively small ($25\times100+100\times50+50\times25+25=8775$ parameters, if I'm not mistaken), these checks won't take you much time. Every time I mention retraining in the following tips, I mean "retraining from scratch", i.e., after weight initialisation, not from the current weights.

• create new training/test sets with a different random split, possibly having the same class ratio for train & test as in the whole dataset ($\frac{250}{150}=\frac{5}{3}$) and retrain

• use a different estimate of generalization error, e.g., $k-$fold cross-validation instead than train/validation/test split. This time you will need to train a few (exactly $k$) times, so this check is a bit more time consuming than the others.

• verify that all data preprocessing steps (apparently, MaxMinScaler only, in your case), have been fit on the training set and applied to the test set, without refitting them on the test set

• you already checked that your problem is actually super-easy: both linear SVM and logistic regression could solve it, so that's an excellent reason to suspect that no overfitting to the test is occuring. It's just an easy problem. Lucky you :-)

• I would have a look at decision regions for a few pairs or triplets of variables, maybe the most influential ones according to linear SVM or logreg. Maybe you'll find out that your problem is (close to) linearly separable, so basically most classifiers will do a great job here.

• if you still are worried, shuffle class labels and retrain. Now the only way for your neural network to get high training set accuracy is to memorize the training set, which will manifest in much longer training time. At the same time, the test set will go down dramatically. If this doesn't happen, there's something seriously wrong somewhere in your code.

• Perform the opposite test: initialise weights and train on just two or three data points. This time, train accuracy will immediately go to 100%, but test set accuracy will stay extremely low, no matter how long you train. If this doesn't happen, again you have a serious bug somewhere.

If at this point you're still worried (you shouldn't), get new test points (which you've never seen until now) and test on them.

Another possibility may be to just scrap the NN and use the linear SVM: you gain

• much faster training & inference
• better interpretability
• convex optimization, i.e., no doubts about the number of training epochs
• cross-validation becomes very simple and feasible
• online training become fast and simple. Well, actually with such a small neural network you could easily perform online training with the NN too.

Answer 2

Most likely you are overfitting your data.

The accuracy you see at the end of your training procedure doesn't tell you much about whether your model is any good. You can make sufficiently complex models fit completely arbitrary training data without actually learning anything whatsoever. To evaluate whether your model is learning something interesting, you want to test it on the testing data (which I assume you have, from your variable names).

In general, choosing the size, number and form of hidden layers is a difficult problem which depends on the type and amount of data you have, as well as the computational resources available to you. A lot of trial and error usually goes into finding a good architecture.

Now specifically to your questions

1. Probably not, as I pointed out above.

2. In principle yes, but in your case it doesn't seem to help very much.

3. Going by those principles, you're doing it wrong, yes. While there can be cases where using larger hidden layers can be useful, you usually come across those after having tried smaller layers.

4. Since you don't have much data but a lot of dimensions, I would expect that you would perform quite well already with only one or two layers. Start trying one layer, play around with different sizes, then try two and see if you actually improve.

5. As you can probably judge from the answer so far, yes, you probably should reduce the number of neurons.

Answer 3

I personally use the following heuristic to choose the number of hidden neurons. Say the input layer has n variables and the output layer has m.

Find the prime number factors of n:

Then find the prime number decomposition of m:

The right amount of hidden neurons that don't overfit usually is among the combinations of these prime factors that divide both the number of input variables and output ones.

For example say you have 520 input features in a multi class classification task with 120 classes. Then

then you should try the following combinations:

 2
 4
 5
 8
10
20
40

In the specific case I was working on the right combination was 40 hidden neurons but of course that depends on the case you are dealing with.