Training error not decreasing on the training set

Question

I cannot make my neural network - MLP with 1 hidden layer fit the training data perfectly. Here is the data:

xs1 = c(-1, 0, 1)
ys1 = c(-0.2445248, 0.1232554, 0.1713998)

This was actually generated by $y = sin(x) + \mathcal{N}(0, 0.25)$ and scaled. The idea is to predict ys1 perfectly from xs1 for the training set.

I define an MLP with 1 hidden layer, 10 neurons:

model <- keras_model_sequential()
model %>% 
  layer_dense(10, input_shape = c(1)) %>%
  layer_dense(activation = "linear", units = 1)

model %>% compile(loss = 'mean_squared_error', optimizer = optimizer_adam(lr = 0.00001))
model %>% fit(Xtrain, Ytrain, batch_size = 32, epochs = 10000, verbose = 2)

So, I would like to have 0 error on this training set. However, this is how convergence looks like:

Epoch 1/10000
 - 2s - loss: 0.0084
Epoch 2/10000
 - 0s - loss: 0.0084
Epoch 3/10000
 - 0s - loss: 0.0083
Epoch 4/10000
 - 0s - loss: 0.0082
Epoch 5/10000
 - 0s - loss: 0.0081
Epoch 6/10000
 - 0s - loss: 0.0081
Epoch 7/10000
 - 0s - loss: 0.0080
Epoch 8/10000
 - 0s - loss: 0.0079
Epoch 9/10000
 - 0s - loss: 0.0079
Epoch 10/10000
 - 0s - loss: 0.0078
Epoch 11/10000
 - 0s - loss: 0.0077
Epoch 12/10000
 - 0s - loss: 0.0077
Epoch 13/10000
 - 0s - loss: 0.0076
Epoch 14/10000
 - 0s - loss: 0.0075
Epoch 15/10000
 - 0s - loss: 0.0075
Epoch 16/10000
 - 0s - loss: 0.0074
Epoch 17/10000
 - 0s - loss: 0.0074
Epoch 18/10000
 - 0s - loss: 0.0073
Epoch 19/10000
 - 0s - loss: 0.0073
Epoch 20/10000
 - 0s - loss: 0.0072
Epoch 21/10000
 - 0s - loss: 0.0071
Epoch 22/10000
 - 0s - loss: 0.0071
Epoch 23/10000
 - 0s - loss: 0.0070
Epoch 24/10000
 - 0s - loss: 0.0070
Epoch 25/10000
 - 0s - loss: 0.0070
Epoch 26/10000
 - 0s - loss: 0.0069
Epoch 27/10000
 - 0s - loss: 0.0069
Epoch 28/10000
 - 0s - loss: 0.0068
Epoch 29/10000
 - 0s - loss: 0.0068
Epoch 30/10000
 - 0s - loss: 0.0067
Epoch 31/10000
 - 0s - loss: 0.0067
Epoch 32/10000
 - 0s - loss: 0.0067
Epoch 33/10000
 - 0s - loss: 0.0066
Epoch 34/10000
 - 0s - loss: 0.0066
Epoch 35/10000
 - 0s - loss: 0.0066
Epoch 36/10000
 - 0s - loss: 0.0065
Epoch 37/10000
 - 0s - loss: 0.0065
Epoch 38/10000
 - 0s - loss: 0.0065
Epoch 39/10000
 - 0s - loss: 0.0064
Epoch 40/10000
 - 0s - loss: 0.0064
Epoch 41/10000
 - 0s - loss: 0.0064
Epoch 42/10000
 - 0s - loss: 0.0064
Epoch 43/10000
 - 0s - loss: 0.0063
Epoch 44/10000
 - 0s - loss: 0.0063
Epoch 45/10000
 - 0s - loss: 0.0063
Epoch 46/10000
 - 0s - loss: 0.0063
Epoch 47/10000
 - 0s - loss: 0.0062
Epoch 48/10000
 - 0s - loss: 0.0062
Epoch 49/10000
 - 0s - loss: 0.0062
Epoch 50/10000
 - 0s - loss: 0.0062
Epoch 51/10000
 - 0s - loss: 0.0061
Epoch 52/10000
 - 0s - loss: 0.0061
Epoch 53/10000
 - 0s - loss: 0.0061
Epoch 54/10000
 - 0s - loss: 0.0061
Epoch 55/10000
 - 0s - loss: 0.0061
Epoch 56/10000
 - 0s - loss: 0.0061
Epoch 57/10000
 - 0s - loss: 0.0060
Epoch 58/10000
 - 0s - loss: 0.0060
Epoch 59/10000
 - 0s - loss: 0.0060
Epoch 60/10000
 - 0s - loss: 0.0060
Epoch 61/10000
 - 0s - loss: 0.0060
Epoch 62/10000
 - 0s - loss: 0.0060
Epoch 63/10000
 - 0s - loss: 0.0060
Epoch 64/10000
 - 0s - loss: 0.0059
Epoch 65/10000
 - 0s - loss: 0.0059
Epoch 66/10000
 - 0s - loss: 0.0059
Epoch 67/10000
 - 0s - loss: 0.0059
Epoch 68/10000
 - 0s - loss: 0.0059
Epoch 69/10000
 - 0s - loss: 0.0059
Epoch 70/10000
 - 0s - loss: 0.0059
Epoch 71/10000
 - 0s - loss: 0.0059
Epoch 72/10000
 - 0s - loss: 0.0059
Epoch 73/10000
 - 0s - loss: 0.0058
Epoch 74/10000
 - 0s - loss: 0.0058
Epoch 75/10000
 - 0s - loss: 0.0058
Epoch 76/10000
 - 0s - loss: 0.0058
Epoch 77/10000
 - 0s - loss: 0.0058
Epoch 78/10000
 - 0s - loss: 0.0058
Epoch 79/10000
 - 0s - loss: 0.0058
Epoch 80/10000
 - 0s - loss: 0.0058
Epoch 81/10000
 - 0s - loss: 0.0058
Epoch 82/10000
 - 0s - loss: 0.0058
Epoch 83/10000
 - 0s - loss: 0.0058
Epoch 84/10000
 - 0s - loss: 0.0058
Epoch 85/10000
 - 0s - loss: 0.0058
Epoch 86/10000
 - 0s - loss: 0.0058
Epoch 87/10000
 - 0s - loss: 0.0058
Epoch 88/10000
 - 0s - loss: 0.0058
Epoch 89/10000
 - 0s - loss: 0.0057
Epoch 90/10000
 - 0s - loss: 0.0057
Epoch 91/10000
 - 0s - loss: 0.0057

As soon as it reaches 0.0057 it does not move lower. Even if run for 10000 iterations.

I tried to change the learning step from 0.1 to 0.0000001 to no avail - still no progress when it hits 0.0057.

If I change the number of layers or neurons - it simply stops at some other value, but never reaches near 0 (for 2 datapoints it actually produces values around $10^{-15}$). I tried different activations - no use. Still not 0.

Can it be local minimum somehow? But it is only 3 points and 10 hidden units - isn't it a very simple surface that should be easily optimized up to very small values?

What am I doing wrong? Can you get 0 on this dataset with this MLP?

Answer 1

layer dense method in keras library uses linear activation by default, which means between the hidden and output layers, you don't accommodate nonlinear transformations. If you stack layers without nonlinearity in-between them, your neural network will be equivalent to a one-layer neural network. Please check here for a simple explanation if you're interested.

So, your neural network is actually one-layer, and that is the output neuron, which means we're trying to solve $wx+b=y$. You have the following least squares problem, $Xa=y:$ $$\begin{bmatrix}-1 & 1\\0 & 1 \\ 1 & 1\end{bmatrix}\begin{bmatrix}w \\ b\end{bmatrix}=\begin{bmatrix}-0.2445248 \\ 0.1232554 \\ 0.1713998\end{bmatrix}$$ The solution to this linear system is $\hat{a}=(X^TX)^{-1}X^Ty$. Mean squared error is $\text{MSE}=\frac{1}{3}||y-\hat{y}||^2$, where $\hat{y}=X\hat{a}=X(X^TX)^{-1}X^Ty$. When you calculate this, you'll obtain $0.0057$, which is actually your global optimum.

Now, put an activation function in the first layer, and watch the system converge to a better optimum.