I'm quite new to neural networks and currently I'm trying to train a non convolutional neural network on the MNIST data set. I'm observing some behaviour I don't quite understand.
This is the code written with keras as the library:
# the data, split between train and test sets (x_train, y_train), (x_test, y_test) = fashion_mnist.load_data() y_train = keras.utils.to_categorical(y_train, 10) y_test = keras.utils.to_categorical(y_test, 10) model = Sequential() early_stopper = EarlyStopping(patience=3) model.add(Flatten(input_shape=(28,28))) model.add(Dense(128, activation="relu")) model.add(Dense(128, activation="relu")) model.add(Dense(10, activation="softmax")) model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"]) model.fit(x_train, y_train, epochs=20, validation_data=(x_test, y_test), callbacks=[early_stopper])
This gives me a validation_acc of around 20%. The funny thing is, when I change the activation functions to "sigmoid", the loss function to "mean_squared_error" and the optimizer to "sgd" my performance improves to around 85% after 50 epochs.
Having read http://neuralnetworksanddeeplearning.com/ I wonder what's the reason for the bad performance of the network I presented in the code. ReLU, cross-entropy and a dynamic optimizer like Adam all seem to improve on the idea of a very vanilla neural network with stochastic gradient optimization, mean squared error as loss and sigmoid activation functions. Yet I get a really bad performance and if I increase the number of nodes in the hidden layers I often get a network that doesn't learn at all.
EDIT: I figured out it has something to do with me not normalizing the input to values between 0 and 1 ... but why is this the problem?