I'm working on a disease classification dataset which has 25 features including the class attribute. It is a binary classification problem. The dataset has total 300 training instances.
I trained a feed-forward neural network with the following architecture:
model = Sequential() model.add(Dense(100, input_dim=train_x.shape, activation='relu')) model.add(Dense(50, activation='relu')) model.add(Dense(25, activation='relu')) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) model.fit(train_x.values, train_y.values, epochs=50, batch_size=40, validation_split=.2, verbose=0)
The architecture has 3 hidden layers with 100, 50 and 25 hidden neurons in the layers.
After training, the model shows the following scores on the test set which includes 100 test instances.
100/100 [==============================] - 0s 94us/step Test loss score: 0.02940008039586246 Test accuracy: 100.0
So, I've got 100% accuracy with this architecture.
The number of hidden layers and hidden neurons per layer was taken arbitrarily.
However, I have found something in an article of Jeff Heaton that:
and also found some rules about choosing the number of hidden neurons per layer:
1. The number of hidden neurons should be between the size of the input layer and the size of the output layer.
2. The number of hidden neurons should be 2/3 the size of the input layer, plus the size of the output layer.
3. The number of hidden neurons should be less than twice the size of the input layer.
These three rules provide a starting point for you to consider.
Now my question is:
Is my model architecture acceptable?
According to Heaton, we can have more than 2 hidden layers for sort of automatic feature engineering. So is using 3 layers okay or not?
Here, I've used 100, 50 and 25 neurons in the hidden layers arbitrarily. The output layer contains only 1 neuron as it is a binary classification. But according to the thumb rule, the number of hidden neurons should be between the size of the input layer and the size of the output layer. So, according to this rule, the number of hidden neurons for this dataset should be between 1-24 as there are 24 input features and the 1 output feature. So, using 100, 50, 25 hidden neurons is wrong?
For this dataset of 300 instances with 24 features and a class attribute of a binary class, what should be the number of hidden neurons? Is there any other empirical rule that I can follow?
Using 100, 50 and 25 hidden neurons yields a good result. So should I decrease the number of neurons? Why?
scikit-learn, can't remember the exact method now but google and you'll find it easily. Also, you've got a small data set: it won't take time (and if it does, just use a GPU on Google Colab). The fact that the SVM is killing this data set confirms my intuition: it's just that your problem is simple. If you had had issues with fitting the logistic 1/ $\endgroup$