Backpropagation demo gives always the same result after training

Question

Just for practice, i've tried to create a super-basic neural network script (one neuron for each layer, didn't want to deal with matrices for the sake of simplicity, not that I don't know how to). I've implemented a squared error loss function, sigmoid activation and the derivatives. Of course i created a feedforward function, and a backprop function to calculate the deltas.

import random
import math

LEARNING_RATE = 0.1
TRAINING_EPOCHS = 10_000  #just a reason to make it incompatible with python2

def sigmoid(x):
    return 1/(1 + math.exp(-x))   #activation

def der_sigmoid(x):
    return x * (1 - x)          #derivative for already applied activation

def cost(x):
    return x * x          #squared error

def der_cost(x):
    return 2 * x          #derivative of squared error


w1 = 2 * random.random() - 1
w2 = 2 * random.random() - 1      #initializing weights

b1 = 0       #biases
b2 = 0

def feedforward(x, w1, w2, b1, b2):
    h = sigmoid((x * w1) + b1) 
    y = sigmoid((h * w2) + b2)        #feedforward

    return h, y

def backprop(q_set, a_set, w1, w2, b1, b2):
    rand_choice = random.randint(0, len(q_set) - 1)

    curr_q = q_set[rand_choice]            #choosing random pair from dataset
    curr_a = a_set[rand_choice]

    h, yHat = feedforward(curr_q, w1, w2, b1, b2)       #calculating guess

    E = yHat - curr_a     #Error

    loss = cost(E)     #Calculating cost, just because
    #print(loss)

    gradient_y = der_cost(E)      #calculating the derivative of the cost for the output layer

    gradient_h = gradient_y * w2   #backpropagating the derivative

    delta_w2 = gradient_y * der_sigmoid(yHat) * w2 * LEARNING_RATE
    delta_w1 = gradient_h * der_sigmoid(h) * w1 * LEARNING_RATE       #calculating weight deltas

    delta_b2 = gradient_y * der_sigmoid(yHat) * LEARNING_RATE         #bias deltas
    delta_b1 = gradient_h * der_sigmoid(h) * LEARNING_RATE

    return delta_b1, delta_b2, delta_w1, delta_w2

set_q = [1,2,3,4,5,6,7,8,9]   
set_a = [0,1,0,1,0,1,0,1,0]     #1 for even, 0 for odd

for i in range(len(set_q)):
    set_q[i] /= max(set_q)     #normalizing the qset between 0 and 1

print(feedforward(5, w1, w2, b1, b2))
print(feedforward(16, w1, w2, b1, b2))        #testing initial predictions

for i in range(TRAINING_EPOCHS):
    delta_b1, delta_b2, delta_w1, delta_w2 = backprop(set_q, set_a, w1, w2, b1, b2)

    w1 -= delta_w1
    w2 -= delta_w2       #applying deltas

    b1 -= delta_b1
    b2 -= delta_b2

print(feedforward(5, w1, w2, b1, b2))       #testing
print(feedforward(16, w1, w2, b1, b2))      #why is this output almost the same?

and, as you already guessed, my problem is that even after training, this pseudo-network doesn't converge, instead it keeps giving the same result (or almost) for any input. What's my mistake? (of course I already tried with various parameters and datasets, but still nothing) Thank you in advance

Answer 1

Your gradient calculations are probably wrong. Let the forward pass be: $$\begin{align}h&=\sigma(wx_1+b_1)\\y&=\sigma(hw_2+b_2)\\L&=(y-a)^2=E^2\end{align}$$ Then, the derivatives wrt $w_i$ is going to be: $$\begin{align}\frac{dL}{dw_1}&=\frac{\partial L}{\partial y}\frac{\partial y}{\partial w_2}=\underbrace{2Ey(1-y)}_{\delta_2}h\\\frac{d L}{d w_2}&=\frac{\partial L}{\partial y}\frac{\partial y}{\partial h}\frac{\partial h}{\partial w_1}=2Ey(1-y)w_2h(1-h)x=\delta_2w_2h(1-h)x\end{align}$$

I couldn't see this calculation in your code.