0
$\begingroup$

I'm working on a disease classification dataset which has 25 features including the class attribute. It is a binary classification problem. The dataset has total 300 training instances.

I trained a feed-forward neural network with the following architecture:

model = Sequential()

model.add(Dense(100, input_dim=train_x.shape[1], activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(25, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(train_x.values, train_y.values, epochs=50, batch_size=40, validation_split=.2, verbose=0)

The architecture has 3 hidden layers with 100, 50 and 25 hidden neurons in the layers.

After training, the model shows the following scores on the test set which includes 100 test instances.

100/100 [==============================] - 0s 94us/step
Test loss score: 0.02940008039586246
Test accuracy: 100.0

So, I've got 100% accuracy with this architecture.

The number of hidden layers and hidden neurons per layer was taken arbitrarily.

However, I have found something in an article of Jeff Heaton that:

enter image description here

and also found some rules about choosing the number of hidden neurons per layer:

1. The number of hidden neurons should be between the size of the input layer and the size of the output layer.

2. The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.

3. The number of hidden neurons should be less than twice the size of the input layer.

These three rules provide a starting point for you to consider.

Now my question is:

  1. Is my model architecture acceptable?

  2. According to Heaton, we can have more than 2 hidden layers for sort of automatic feature engineering. So is using 3 layers okay or not?

  3. Here, I've used 100, 50 and 25 neurons in the hidden layers arbitrarily. The output layer contains only 1 neuron as it is a binary classification. But according to the thumb rule, the number of hidden neurons should be between the size of the input layer and the size of the output layer. So, according to this rule, the number of hidden neurons for this dataset should be between 1-24 as there are 24 input features and the 1 output feature. So, using 100, 50, 25 hidden neurons is wrong?

  4. For this dataset of 300 instances with 24 features and a class attribute of a binary class, what should be the number of hidden neurons? Is there any other empirical rule that I can follow?

  5. Using 100, 50 and 25 hidden neurons yields a good result. So should I decrease the number of neurons? Why?

$\endgroup$
  • 1
    $\begingroup$ Don’t pay attention to Heaton’s blog and focus on your problem. The fact that you get such an high accuracy without any regularization at all is indeed remarkable. 1) what's the prevalence of the minority class in the training and in the test set? 2) Try applying a linear SVM to your data with & without regularization and tell us what you get. I have other suggestions which I may post as an answer, but for now please check these. $\endgroup$ – DeltaIV Jun 11 '18 at 6:56
  • 1
    $\begingroup$ In the training set, there are 300 instances in which 183 are YES and 117 are NO class. In the training set, there are 100 instances in which 67 are YES and 33 are NO class. I've just used MaxMinScaler to the attributes. I've trained a linear SVC with 'l2' and without any regularization. Both of them yields 100% accuracy. I've also trained a logistic regression which yields 98% accuracy. $\endgroup$ – Abdullah Al Imran Jun 11 '18 at 7:45
  • $\begingroup$ You meant "in the test set there are 100 instances", but ok. The NO group is a bit underrepresented in the test set: nothing too serious, but just for pedantry consider splitting again train set & test set in a balanced way: it's trivial with scikit-learn, can't remember the exact method now but google and you'll find it easily. Also, you've got a small data set: it won't take time (and if it does, just use a GPU on Google Colab). The fact that the SVM is killing this data set confirms my intuition: it's just that your problem is simple. If you had had issues with fitting the logistic 1/ $\endgroup$ – DeltaIV Jun 11 '18 at 9:22
  • 1
    $\begingroup$ @DeltaIV Thanks for your comment. I am actually trying various algorithms on this dataset. Finally, I'll compare the performance of the classifiers and choose the best one. NN could be a bad method for this problem. But, I just want to make sure that I've tried a logically acceptable NN architecture. $\endgroup$ – Abdullah Al Imran Jun 11 '18 at 11:54
  • 1
    $\begingroup$ You're welcome, but please note I didn't say anything about NNs being bad. Any method which gets you 100% test accuracy is not bad by definition, provided you have convincingly ruled out any overfitting to the test set. I was just saying that with such a simple problem, SVMs provide a few advantages over NNs, such as: faster training/inference time, convex optimization (so no need to bother with early stopping, validation sets, etc.), easily interpretable (you said it's a linear SVM, right?) and online training becomes very easy if you get new data (a bit unlikely in your case, I know). $\endgroup$ – DeltaIV Jun 11 '18 at 13:34
3
$\begingroup$

In Deep Learning there are no hard & fast rules to set the number of layers, the number of hidden units per layer and not even the kind of connections between layers: who claims the contrary often doesn't have experience of struggling with modern architectures, and relies on insight 20 years (if not more) old. "Proofs" of my claim:

  • you have all kinds of different architectures performing increasingly better on ImageNet: VGG, Inception, ResNet, ResNext, Xception, DenseNet, etc. Even the simplest empirical rules such as "double the number of channels before pooling" or even just "use pooling after a convolutional layer" aren't valid in general, though replacing pooling with increasing strides in successive layers has its drawbacks

  • if the "back-of-the-envelope" rules to choose the number of hidden layers & neurons from the days of old were still valid today, we wouldn't have so much investment on automated architecture learning - I think after Deep RL and GANs, this is currently the single biggest R&D investment in AI companies.

But that's not necessarily bad news: your architecture is getting excellent test set accuracy as it is, so maybe you just don't have to worry about optimal architecture. Accuracy is not the correct metric to look at for classification, but as you have nearly balanced classes, and anyway astonishing classification accuracy, it's not like using another metric will make a big difference.

Since you haven't provided some data, or some training curves, it's hard to say if overfitting to the training set is actually happening. However, there are a few checks you could make, to clear any doubts. Since your architecture is relatively small ($25\times100+100\times50+50\times25+25=8775$ parameters, if I'm not mistaken), these checks won't take you much time. Every time I mention retraining in the following tips, I mean "retraining from scratch", i.e., after weight initialisation, not from the current weights.

  • create new training/test sets with a different random split, possibly having the same class ratio for train & test as in the whole dataset ($\frac{250}{150}=\frac{5}{3}$) and retrain

  • use a different estimate of generalization error, e.g., $k-$fold cross-validation instead than train/validation/test split. This time you will need to train a few (exactly $k$) times, so this check is a bit more time consuming than the others.

  • verify that all data preprocessing steps (apparently, MaxMinScaler only, in your case), have been fit on the training set and applied to the test set, without refitting them on the test set

  • you already checked that your problem is actually super-easy: both linear SVM and logistic regression could solve it, so that's an excellent reason to suspect that no overfitting to the test is occuring. It's just an easy problem. Lucky you :-)

  • I would have a look at decision regions for a few pairs or triplets of variables, maybe the most influential ones according to linear SVM or logreg. Maybe you'll find out that your problem is (close to) linearly separable, so basically most classifiers will do a great job here.

  • if you still are worried, shuffle class labels and retrain. Now the only way for your neural network to get high training set accuracy is to memorize the training set, which will manifest in much longer training time. At the same time, the test set will go down dramatically. If this doesn't happen, there's something seriously wrong somewhere in your code.

  • Perform the opposite test: initialise weights and train on just two or three data points. This time, train accuracy will immediately go to 100%, but test set accuracy will stay extremely low, no matter how long you train. If this doesn't happen, again you have a serious bug somewhere.

If at this point you're still worried (you shouldn't), get new test points (which you've never seen until now) and test on them.

Another possibility may be to just scrap the NN and use the linear SVM: you gain

  • much faster training & inference
  • better interpretability
  • convex optimization, i.e., no doubts about the number of training epochs
  • cross-validation becomes very simple and feasible
  • online training become fast and simple. Well, actually with such a small neural network you could easily perform online training with the NN too.
$\endgroup$
1
$\begingroup$

Most likely you are overfitting your data.

The accuracy you see at the end of your training procedure doesn't tell you much about whether your model is any good. You can make sufficiently complex models fit completely arbitrary training data without actually learning anything whatsoever. To evaluate whether your model is learning something interesting, you want to test it on the testing data (which I assume you have, from your variable names).

In general, choosing the size, number and form of hidden layers is a difficult problem which depends on the type and amount of data you have, as well as the computational resources available to you. A lot of trial and error usually goes into finding a good architecture.

Now specifically to your questions

  1. Probably not, as I pointed out above.

  2. In principle yes, but in your case it doesn't seem to help very much.

  3. Going by those principles, you're doing it wrong, yes. While there can be cases where using larger hidden layers can be useful, you usually come across those after having tried smaller layers.

  4. Since you don't have much data but a lot of dimensions, I would expect that you would perform quite well already with only one or two layers. Start trying one layer, play around with different sizes, then try two and see if you actually improve.

  5. As you can probably judge from the answer so far, yes, you probably should reduce the number of neurons.

$\endgroup$
  • 1
    $\begingroup$ The accuracy and loss I showed above was the testing accuracy & loss. $\endgroup$ – Abdullah Al Imran Jun 11 '18 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.