0
$\begingroup$

I'm trying to implement a neural network in sklearn. I'm using stochastic gradient descent ('sgd') as the solver, an activation function of 'tanh' and all other values as the default ones provided by the library. I'm varying the value of hidden_layer_size from (10,1) to (100, 30) and noting the f1 score returned by 10- fold cross validation to find the optimal number of layers to keep in the model. However, my f1 scores continue to be constant at 0.8961, regardless of what the hidden layer size is.

Is this expected behavior?

$\endgroup$
  • $\begingroup$ Check if the data is properly balanced between various classes. Check the actual predicted label. Sometimes an invariant classification metric arises from all inputs being assigned the same output class and therefore the TP, TN, FP, FN don't change. What are the number of epochs and learning rate you are using? Please provide more relevant info. $\endgroup$ – Dynamic Stardust Dec 14 '17 at 22:50
0
$\begingroup$

1) Please use minibatch if your dataset is of medium size or full batch if you can afford or your datasize is smaller. In my view SGD is used in case when datasize is very huge.

2) Repalce tanh with ReLu. tanh and sigmoid saturates learning. That could be the case not achieving better accuracy.

3) Theoretically, Accuracy should increase by increasing the number of layers and layer sizes but in practice that doesn't happen because of cost of train deep network also increases. To solve that, "ResNet" (Residual Network architecture) can be used where you need to pass on the feedback from the previous to previous layer to the current layer also. I think, this can be tried when the above two doesn't work well.

4) You might have achieved the best numbers there is nothing left by adding more layers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.