I wanted to see if a neural network could learn the identity function using the MNIST handwritten dataset.
Here is the full code
import keras from keras.datasets import mnist from keras.models import Sequential from keras.layers import Dense from keras.optimizers import RMSprop batch_size = 128 epochs = 20 (x_train, y_train), (x_test, y_test) = mnist.load_data() x_train = x_train.reshape(60000, 784) x_test = x_test.reshape(10000, 784) model = Sequential() model.add(Dense(784, activation='relu', input_shape=(784,))) model.add(Dense(784, activation='relu')) model.add(Dense(784, activation='relu')) model.add(Dense(784, activation='relu')) model.summary() model.compile(loss='mean_squared_error', optimizer=RMSprop(), metrics=['mean_absolute_percentage_error']) history = model.fit(x_train, x_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_data=(x_test, x_test)) score = model.evaluate(x_test, x_test, verbose=0) print('Test loss:', score) print('Test MAPE:', score)
and the output
**4 dense layers** Epoch 20/20 60000/60000 [==============================] - 50s 840us/step - loss: 456.7581 - mean_absolute_percentage_error: 351097677.7045 - val_loss: 523.7151 - val_mean_absolute_percentage_error: 504905991.0656 Test loss: 523.7150838867187 Test MAPE: 504905988.5056
What I can't quite get my head around is why training cannot find the perfect solution to the problem and why it takes so long to even get close to it? Even with one dense layer the exact solution cannot be found:
**1 dense layer** Epoch 20/20 60000/60000 [==============================] - 16s 268us/step - loss: 180.6187 - mean_absolute_percentage_error: 209296481.2373 - val_loss: 167.9543 - val_mean_absolute_percentage_error: 192590419.9936 Test loss: 167.954341796875 Test MAPE: 192590420.1984
Conceptually I can see that there is a solution space (not just the exact identity function) as it is likely there are some pixels that have the same value as each other in all the images which could be swapped in the training set at no loss (0's around the edge for example). With the knowledge that this is a local minimum can I learn anything from this to guide me out rather than playing with hyperparameters until I find something better?