# Backpropagation in multi-layer perceptron (MLP) doesn't converge [closed]

My simple fully-connected multilayer perceptron (MLP) that I'm writing for academic purposes is causing to me sleep deprivation.

I can't figure out why my MLP learns poorly, even if I try to solve monk's problems where performance (in terms of accuracy) is well understood.

I'm just wondering if it's the way the backpropagation algorithm is implemented the real cause.

The code is plain simple, here I give the main functions implementing the backprop.

Here the forward function:

def forward(self, x, y, n):
"""
x: the input vector representig the input signal
y: the target vector, the true output
n: the lenght of vectors x and y
"""
o = self.propagateSignal(x)
loss = self.objective.apply(y, o)/n
penalty = self._lambda*np.sum([np.sum(np.linalg.norm(layer.W, 2)**2) for layer in self.nnet])
return (loss + penalty)


where propagateSignal is defined as:

def propagateSignal(self, x):
_h = x
for layer in self.nnet:
_h = layer.apply(_h)
return _h


Then the chain-rule pass implemented by the backward function

def backward(self, y, n):
# after the forward computation, compute the gradient on the output layer
o = self.nnet[-1].h # last layer's output
for layer, grad in zip(reversed(self.nnet), reversed(self.gradients)): # starting backward we reconstruct error with respect to each weight
# convert the gradient on the layer's output into a gradient into
# the pre-nonlinearity activation (element-wise (hadamard) multiplication if f is elementwise)
g = g*layer.activation.dh(layer.a)
# compute gradients on weights and biases (including the regularization term,
# where needed):
grad.nablaW += np.outer(g, layer._h) + 2*self._lambda*layer.W
# propagate the gradients w.r.t. the next lower-level hidden layer's activations
g = np.dot(layer.W.T, g)


and finally the descent function used to update weights:

def descent(self):
layer.W = layer.W - nablaW_new
layer.b = layer.b - nablab_new
# we clear-up the sum of gradients with respect the last seen example in the case of online mode or
# in the last epoch (complete pass on dataset) for batch-mode


[EDIT]:

For the sake of clarity I pair code with math. As I've already pointed out the math is taken from "Deep Learning" book by Goodfellow-Bengio-Courville at pages 208-209 of Chapter 6.

Let's see first what happens in the forward phase.

Let $W^{(i)}$ and $b^{(i)}$, $i\in\{1,...,l\}$, be the weight matrices and bias vectors for each layer of the model respectively. Let also $x$ be the input signal and $y$ the target output.

Let $h^{(0)} = x$, then $\forall~ k = 1,..., l$:

$$a^{(k)} = b^{(k)} + W^{(k)}h^{k-1}$$

$$h^{(k)} = f(a^{(k)})$$

def propagateSignal(self, x):
_h = x
for layer in self.nnet:
_h = layer.apply(_h)
return _h


Let $\hat{y} = h^{(l)}$ be the last layer's output, then the total cost function could be computed as follows

$$J = L(y, \hat{y}) + \lambda\Omega(\theta)$$

where $L$ is the loss function added to a regularizer $\Omega(\theta)$ defined over the network's free variables (weights and biases).

def forward(self, x, y, n):
"""
x: the input vector representig the input signal
y: the target vector, the true output
n: the lenght of vectors x and y
"""
o = self.propagateSignal(x)
loss = self.objective.apply(y, o)/n
penalty = self._lambda*np.sum([np.sum(np.linalg.norm(layer.W, 2)**2 + np.linalg.norm(layer.b))**2 for layer in self.nnet])
return (loss + penalty)


Let's look then at the backward part.

After the forward computation, compute the gradient on the output layer:

$g \leftarrow \nabla_{\hat{y}}J = \nabla_{\hat{y}}L(y, \hat{y})$

o = self.nnet[-1].h # last layer's output


Then $\forall~k=l,l-1,...,1:$

Convert the gradient on the layer’s output into a gradient into the pre-nonlinearity activation (element-wise multiplication if f is element-wise):

$g \leftarrow \nabla_{a^{(k)}}J = g\odot f'(a^{(k)})$

g = g*layer.activation.dh(layer.a)


Compute gradients on weights and biases (including the regularization term,where needed):

$\nabla_{b^{(k)}} J = g + \lambda\nabla_{b^{(k)}}\Omega(\theta)$

grad.nablab += g + 2*self._lambda*layer.b


$\nabla_{W^{(k)}} J = gh^{(k-1)T} + \lambda\nabla_{W^{(k)}}\Omega(\theta)$

grad.nablaW += np.outer(g, layer._h) + 2*self._lambda*layer.W


Propagate the gradients w.r.t. the next lower-level hidden layer’s activations

$g \leftarrow \nabla_{h^{(k-1)}}J = W^{(k)T}g$

Weights are then updated using descent function as showed at the beginning.

[EDIT]:

I don't know if the math I used and its implementation is correct or not. Trying to solve classification tasks given in "The Monk's Problems" I don't see the expected convergence on test set, that is, for example, a 100% accuracy on Monk 1 problem. What's more strange is that validation/test error is sometimes or always (depending on how weights are initialized?) below the training one.

Hidden layers and output layer have sigmoid activation functions, and I minimize the mean euclidean loss.

So, I want to be sure of what I've implemented.

• What language is that? Matlab? – generic_user Oct 20 '17 at 13:38
• Why don't you write out your algorithm in both math and code, so that we can see what you're trying to represent with your code. – generic_user Oct 20 '17 at 13:39
• Code is written in Python 3. Math for forward and backward algorithms is taken from Goodfellow-Bengio-Courville's "Deep Learning" book at pages 208-209 of Chapter 6. Here you can find a freely available copy deeplearningbook.org/contents/mlp.html – Luigi Oct 20 '17 at 14:20
• On what dataset you don't see convergence? On the training set or on the test set? Also, you definitely shouldn't expect community members to go and read your link, in order to answer your question. Your question should be complete in itself. As it is now, without reading the link we have no idea what the dataset is and what the loss (objective) function is. What should your MLP learn to do? – DeltaIV Oct 20 '17 at 18:40
• I'm just trying to do some prior benchmarking in order to see if my MLP behaves as expected. In solving "The Monk's Problems", that is basically a classification task, I don't see any good convergence on test set where is expected to be a 100% accuracy for Monk Problem 1, mine is nearly 40%. Worst thing is that my MLP exhibit a very queer behavior, validation/test error is under training error curve for any hyper-parameter, even the ones they are considered "correct" for that problem. Hidden layers are basically sigmoid functions as it is the last one (the output). – Luigi Oct 23 '17 at 9:16