# Backpropagation demo gives always the same result after training

Just for practice, i've tried to create a super-basic neural network script (one neuron for each layer, didn't want to deal with matrices for the sake of simplicity, not that I don't know how to). I've implemented a squared error loss function, sigmoid activation and the derivatives. Of course i created a feedforward function, and a backprop function to calculate the deltas.

import random
import math

LEARNING_RATE = 0.1
TRAINING_EPOCHS = 10_000  #just a reason to make it incompatible with python2

def sigmoid(x):
return 1/(1 + math.exp(-x))   #activation

def der_sigmoid(x):
return x * (1 - x)          #derivative for already applied activation

def cost(x):
return x * x          #squared error

def der_cost(x):
return 2 * x          #derivative of squared error

w1 = 2 * random.random() - 1
w2 = 2 * random.random() - 1      #initializing weights

b1 = 0       #biases
b2 = 0

def feedforward(x, w1, w2, b1, b2):
h = sigmoid((x * w1) + b1)
y = sigmoid((h * w2) + b2)        #feedforward

return h, y

def backprop(q_set, a_set, w1, w2, b1, b2):
rand_choice = random.randint(0, len(q_set) - 1)

curr_q = q_set[rand_choice]            #choosing random pair from dataset
curr_a = a_set[rand_choice]

h, yHat = feedforward(curr_q, w1, w2, b1, b2)       #calculating guess

E = yHat - curr_a     #Error

loss = cost(E)     #Calculating cost, just because
#print(loss)

gradient_y = der_cost(E)      #calculating the derivative of the cost for the output layer

delta_w2 = gradient_y * der_sigmoid(yHat) * w2 * LEARNING_RATE
delta_w1 = gradient_h * der_sigmoid(h) * w1 * LEARNING_RATE       #calculating weight deltas

delta_b2 = gradient_y * der_sigmoid(yHat) * LEARNING_RATE         #bias deltas
delta_b1 = gradient_h * der_sigmoid(h) * LEARNING_RATE

return delta_b1, delta_b2, delta_w1, delta_w2

set_q = [1,2,3,4,5,6,7,8,9]
set_a = [0,1,0,1,0,1,0,1,0]     #1 for even, 0 for odd

for i in range(len(set_q)):
set_q[i] /= max(set_q)     #normalizing the qset between 0 and 1

print(feedforward(5, w1, w2, b1, b2))
print(feedforward(16, w1, w2, b1, b2))        #testing initial predictions

for i in range(TRAINING_EPOCHS):
delta_b1, delta_b2, delta_w1, delta_w2 = backprop(set_q, set_a, w1, w2, b1, b2)

w1 -= delta_w1
w2 -= delta_w2       #applying deltas

b1 -= delta_b1
b2 -= delta_b2

print(feedforward(5, w1, w2, b1, b2))       #testing
print(feedforward(16, w1, w2, b1, b2))      #why is this output almost the same?


and, as you already guessed, my problem is that even after training, this pseudo-network doesn't converge, instead it keeps giving the same result (or almost) for any input. What's my mistake? (of course I already tried with various parameters and datasets, but still nothing) Thank you in advance

Your gradient calculations are probably wrong. Let the forward pass be: \begin{align}h&=\sigma(wx_1+b_1)\\y&=\sigma(hw_2+b_2)\\L&=(y-a)^2=E^2\end{align} Then, the derivatives wrt $$w_i$$ is going to be: \begin{align}\frac{dL}{dw_1}&=\frac{\partial L}{\partial y}\frac{\partial y}{\partial w_2}=\underbrace{2Ey(1-y)}_{\delta_2}h\\\frac{d L}{d w_2}&=\frac{\partial L}{\partial y}\frac{\partial y}{\partial h}\frac{\partial h}{\partial w_1}=2Ey(1-y)w_2h(1-h)x=\delta_2w_2h(1-h)x\end{align}