I'm trying to understand the effect of the learning rate on a 10x10x10x10x4 sequential NN. Where each hidden layer is ReLu and the output layer being Softmax.

I know the theory: low rate -> slow convergence, high rate -> chance of overshooting miminum. Yet in practice it seems a bit harder to understand the output.

As data, I generated 10 samples (2-dim points clustered into 4 classes). enter image description here

If I train the NN on the raw data the results hugely depend on the learning rate:

  • [1.] Using a rate of 0.4, 0.1 or 0.01 results in a good accuracy (plot on the left).
  • [2.] Using a rate of 0.5 results in bad accuracy (the middle plot)
  • [3.] Using a rate of 0.001 also result in bad accuracy but seems to have other behavior (plot on the right( I increased the amount of epochs))

enter image description here

Question 1 : What causes this too happen? Is the learning rate of 0.5 too large [2.], resulting in overshooting the minimum? I'm trying to visualize this in my head. In addition, while setting the learning rate to something smaller (eg. 0.001) as shown in [3.] I'd expect slower convergence, but I hardly see anything here (I already increased the amount of epochs). I'm guessing that he predicts all outputs to be of one particular class, but I'm not completely sure why.

Question 2 : Often one needs to standardize the input data. If I do this using sklearn's StandardScaler it however also influences the learning rate's effect. Using a learning rate above 0.1 on this data results in [2.]. Why is this so different?

Question 3 : Since I'm initializing my weights from a normal distribution, some initial weights could be negative. This can result in a negative netinput. Since ReLu isn't activating for negative input. Also the derivative at the negative part is 0, resulting in dead neurons. How does this effect learning? Also standardizing your input results in getting negative input values, doesn't this induce the same problem?

Code for learning

# Parameters
lr = 0.4
input_dim = X_train.shape[1]
activation_hidden = 'relu' 
activation_output = 'softmax'
initializer = keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=None)
SGD = keras.optimizers.SGD(lr=lr)

model = Sequential()
model.add(Dense(10, input_dim=input_dim, kernel_initializer=initializer,activation=activation_hidden))
model.add(Dense(10, kernel_initializer=initializer,activation=activation_hidden))
model.add(Dense(10, kernel_initializer=initializer,activation=activation_hidden))
model.add(Dense(10, kernel_initializer=initializer,activation=activation_hidden))
model.add(Dense(y_train.shape[1], kernel_initializer=initializer,activation=activation_output))


history = model.fit(X_train, y_train, epochs=200, batch_size=64, verbose=1, validation_split=0.2)

marked as duplicate by Reinstate Monica neural-networks Jul 31 at 15:14

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.