0
$\begingroup$

This is a rather long question. Sorry for that. The main thing is that I tried to derive out backpropagation via chain rule etc. I am aware that the index notation changes to transpose the matrix while doing multiplication; and also that for matrix calculus Hadamard product is needed. However, I am not exactly clear where each has to apply. I derived out the equation of the last layer and inner layers with index notation. I tried to implement a simple CNN to test this out; however, during backpropagation, the weights are not matching. Even when I try to do Hadamard product (element-wise multiplication). Not able to break through.

Excerpt from https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/


Neural Network

$$ \mathbf { \bbox[10px, border:2px solid red] { \color{red}{ \begin{aligned} a^0 \rightarrow \bbox[5px, border:2px solid black] { \underbrace{\text{hidden layers}}_{a^{l-2}} } \,\rightarrow \bbox[5px, border:2px solid black] { \underbrace{w^{l-1} a^{l-2}+b^{l-1}}_{z^{l-1} } } \,\rightarrow \bbox[5px, border:2px solid black] { \underbrace{\sigma(z^{l-1})}_{a^{l-1}} } \,\rightarrow \bbox[5px, border:2px solid black] { \underbrace{w^l a^{l-1}+b^l}_{z^{l}/logits } } \,\rightarrow \bbox[5px, border:2px solid black] { \underbrace{P(z^l)}_{\vec P/ \text{softmax} /a^{l}} } \,\rightarrow \bbox[5px, border:2px solid black] { \underbrace{L ( \vec P, \vec Y)}_{\text{CrossEntropyLoss}} } \end{aligned} }}} $$


CrossEntropy Loss with respect to Weight in last layer

$$ \mathbf { \frac {\partial L}{\partial w^l} = \color{red}{\frac {\partial L}{\partial z^l}}.\color{green}{\frac {\partial z^l}{\partial w^l}} \rightarrow \quad EqA1 } $$ $$ \color{red} { \begin{aligned} \frac {\partial L}{\partial z^l} = p_i - y_i \rightarrow \quad \text{EqA1.1} \end{aligned} } $$ We need to put this back in $EqA1$. We now need to calculate the second term, to complete the equation

$$ \begin{aligned} \frac {\partial L}{\partial w^l} = \color{red}{\frac {\partial L}{\partial z^l}}.\color{green}{\frac {\partial z^l}{\partial w^l}} \\ \\ z^{l} = (w^l a^{l-1}+b^l) \\ \color{green}{\frac {\partial z^l}{\partial w^l} = a^{l-1}} \\ \\ \text{Putting all together} \\ \\ \frac {\partial L}{\partial w^l} = (p_i - y_i) *a^{l-1} \quad \rightarrow \quad \mathbf {EqA1} \end{aligned} $$

Using Gradient descent we can keep adjusting the last layer like

$$ w{^l}{_i} = w{^l}{_i} -\alpha * \frac {\partial L}{\partial w^l} $$

Now let's do the derivation for the inner layers

Derivative of Loss with respect to Weight in Inner Layers

he trick here (yes it is a trick), is to derivative the Loss with respect to the inner layer as a composition of the partial derivative we computed earlier. And also to compose each partial derivative as partial derivative with respect to either $z^x$ or $w^x$ but not with respect to $a^x$. This is to make derivatives easier and intuitive to compute.

$$ \begin{aligned} \frac {\partial L}{\partial w^{l-1}} = \color{blue}{\frac {\partial L}{\partial z^{l-1}}}. \color{green}{\frac {\partial z^{l-1}}{\partial w^{l-1}}} \rightarrow \text{EqA2} \end{aligned} $$

The trick is to represent the first part in terms of what we computed earlier; in terms of $\color{blue}{\frac {\partial L}{\partial z^{l}}}$

$$ \begin{aligned} \color{blue}{\frac {\partial L}{\partial z^{l-1}}} = \color{blue}{\frac {\partial L}{\partial z^{l}}}. \frac {\partial z^{l}}{\partial a^{l-1}}. \frac {\partial a^{l-1}}{\partial z^{l-1}} \rightarrow \text{ Eq with respect to Prev Layer} \\ \\ \color{blue}{\frac {\partial L}{\partial z^{l}}} = \color{blue}{(p_i- y_i)} \text{ from the previous layer (from EqA1.1) } \\ \\ z^l = w^l a^{l-1}+b^l \text{ which makes } {\frac {\partial z^{l} }{\partial a^{l-1}} = w^l} \\ \text{ and } a^{l-1} = \sigma (z^{l-1}) \text{ which makes } \frac {\partial a^{l-1}}{\partial z^{l-1}} = \sigma \color{red}{'} (z^{l-1} ) \\ \\ \text{ Putting together we get the first part of Eq A2 } \\\\ \color{blue}{\frac {\partial L}{\partial z^{l-1}}} =\color{blue}{(p_i- y_i)}.w^l.\sigma \color{red}{'} (z^{l-1} ) \rightarrow \text{EqA2.1 } \\ \\ z^{l-1} = w^{l-1} a^{l-2}+b^{l-1} \text{ which makes } \color{green}{\frac {\partial z^{l-1}}{\partial w^{l-1}}=a^{l-2}} \\ \\ \frac {\partial L}{\partial w^{l-1}} = \color{blue}{\frac {\partial L}{\partial z^{l-1}}}. \color{green}{\frac {\partial z^{l-1}}{\partial w^{l-1}}} = \color{blue}{(p_i- y_i)}.w^l.\sigma \color{red}{'} (z^{l-1} ).\color{green}{a^{l-2}} \end{aligned} $$


So far so good; Now here is the code snippet where I try to apply this. For the last layer it works. But for the inner layer it does not match weights. Which means I am somewhere wrong


Excerpt from https://github.com/alexcpn/cnn_in_python

if __name__ == '__main__':
    
    np.set_printoptions(formatter={'float': lambda x: "{0:0.2f}".format(x)})
    # Generate a random image
    image_size = 32 
    image_depth = 3
    image = np.random.rand(image_size, image_size)
    # to mimic RGB channel
    image = np.stack([image,image,image], axis=image_depth-1) # 0 to 2
    print("Image Shape=",image.shape)
    print("Image [0,0,:]=",image[0,1,2])

    # The class containing the convolution Logic
    testConv2D = cnn.Conv2D()

    # we will create leNet without the Pooling parts
    # (stride is always 1 for now)
    #  [32.32.3] *(5.5.3)*6  == [28.28.6] * (5.5.6)*1 = [24.24.1] *  (5.5.3)*16 = [20.20.16] * FC layer 
    
    print("-----------Forward Propagation------------------------")

    # For layer 1
    filter_size = 5  
    number_of_filters = 6
    # Initialize the weight's/filters of Layer1
    w1 = initializeWeights(number_of_filters,filter_size,image.shape[2])
    a1 = layerConvolutionActivation(image, filter_size,number_of_filters,w1)
    
     # For layer 2
    filter_size = 5  
    number_of_filters = 1
    # Initialize the weight's/filters of Layer1
    w2 =  initializeWeights(number_of_filters,filter_size,a1.shape[2])
    # Do convolution and activation
    a2 = layerConvolutionActivation(a1, filter_size,number_of_filters,w2)
    
    # For layer 3
    # Out=W−F+1 imagesize - filtersize + 1
    filter_size = 5  
    number_of_filters = 16
    # Initialize the weight's/filters of Layer1
    w3 =  initializeWeights(number_of_filters,filter_size,a2.shape[2])
    # Do convolution and activation
    a3 = layerConvolutionActivation(a2, filter_size,number_of_filters,w3)
    print("a3.shape=", a3.shape) # output_layer3 shape = (20, 20, 16)
        
    """ 
    Lets add the fully connected layer say 120 - we need the shape to be compatible - for that we are adding the
    the dimension of the above layer 
    """
    w4 =  np.random.rand(a3.shape[0],120,a3.shape[2]) 
    print("w4.shape =", w4.shape) # (20, 120, 16)
    # this time there is no convolution - rather we need to do a dot
    z4 = np.einsum('ijp,jkp->ik', a3, w4) # (20, 120)
    #output_layer4 = np.tensordot(output_layer3,weight_layer4,axes=2)
    print("z4.shape =", z4.shape)
    a4 = util.sigmoid(z4)
    print("a4.shape =", a4.shape)

    w5 =  np.random.rand(a4.shape[1],1) 
    print("w5.shape =", w5.shape) # (20, 120, 16)
    # this time there is no convolution - rather we need to do a dot
    z5 = np.einsum('ij,jk->ik', a4, w5) # (20, 120)
    print("z5.shape =", z5.shape)
    a5 = util.sigmoid(z5)
    print("a5.shape =", a5.shape)
    a5_derv = util.derv_sigmoid(z5)
    print("a5_derv.shape =", a5_derv.shape)

    # final layer lets make it 10 classes
    w6 =  np.random.rand(a5.shape[0],10) 
    print("w6.shape =", w6.shape) # (20, 120, 16)
    # this time there is no convolution - rather we need to do a dot
    logits = np.einsum('ij,ik->jk', a5, w6).flatten() #  20,1*20,10 (1, 10) == Z_l
    z6 = logits
    print("z6.shape=", z6.shape) #(10,)
    #print("Final  Output  =", output_layer6)
    
    """
    Run Softmax
    """
    softmax_ouput =util.softmax(logits)
    a6 = softmax_ouput
    print("a6.shape =", a6.shape)
    print("Softmax Output  =", softmax_ouput)

     # Assume that the truth was class 1 , for this particular "image"
    target = np.array([1., 0., 0., 0., 0. ,0., 0. ,0 ,0., 0.])
     
    #Plug this into the cost function lets take the CrossEntropy Loss as this a classification
    # See this https://www.youtube.com/watch?v=dEXPMQXoiLc  
    # Get index of the true class
    LcrossEntropyLoss = util.crossentropyloss(softmax_ouput,target)
    print("crossEntropyLoss  = ",LcrossEntropyLoss) # ex 3.75671634607845
    
    """
    Back-Propagate the Loss
    """
    print("-----------Back Propagation------------------------")

    lr = 1 # learning rate
    # https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/
    D_S_by_z = util.derv_softmax_wrto_logits(softmax_ouput)
    D_L_by_z = util.derv_crossentropyloss_wrto_logits(softmax_ouput,target)

    # For the last layers  W= 6
    # EqA1 in https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/
    activation_L =  softmax_ouput
    B0 = (activation_L -target)
    print("B0.shape =", B0.shape)
    D_L_by_w6 = B0 *a5
    print("BP: Last weight update - D_L_by_w6 shape==w6 shape",D_L_by_w6.shape,w6.shape)
    w6 = w6 - lr*D_L_by_w6
    

    print("-----------------------------------")
    # For the inner layers  W= 5
    # EqA2 in https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/
    B1 = w6 * B0  
    B1 = B1 @ a5_derv
    print("B1 shape",B1.shape)
    D_L_by_w5 =   a4.T @ B1
    print("D_L_by_w5 shape",D_L_by_w5.shape)
    D_L_by_w5 = np.expand_dims(D_L_by_w5, axis=1)
    print("D_I_by_w5 shape",D_L_by_w5.shape)
    print("w5 shape", w5.shape)
    w5 = w5 - lr*D_L_by_w5
    print("w5 shape", w5.shape)

Output

cnn_1/cnn_py$ python3 main.py 
Image Shape= (32, 32, 3)
Image [0,0,:]= 0.5815285626863921
-----------Forward Propogation------------------------
Convolution Shape = (28, 28, 6)
Convolution Shape = (24, 24, 1)
Convolution Shape = (20, 20, 16)
a3.shape= (20, 20, 16)
w4.shape = (20, 120, 16)
z4.shape = (20, 120)
a4.shape = (20, 120)
w5.shape = (120, 1)
z5.shape = (20, 1)
a5.shape = (20, 1)
a5_derv.shape = (20, 1)
w6.shape = (20, 10)
z6.shape= (10,)
a6.shape = (10,)
Softmax Output  = [0.02 0.26 0.03 0.08 0.09 0.04 0.07 0.01 0.01 0.39]
crossEntropyLoss  =  4.091121759807155
-----------Back Propogation------------------------
1
B0.shape = (10,)
BP: Last weight update - D_L_by_w6 shape==w6 shape (20, 10) (20, 10)
-----------------------------------
Traceback (most recent call last):
  File "/home/alex/coding/cnn_1/cnn_py/main.py", line 172, in <module>
    B1 = B1 @ a5_derv
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 20 is different from 10)

$\endgroup$

1 Answer 1

0
$\begingroup$

The equations were correct. I had done a mistake in the code ; there were some element wise operations and some dot product type operations. It is corrected now - code

   print("-----------Back Propogation------------------------")

    lr = 1 # learning rate
    # https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/
    D_S_by_z = util.derv_softmax_wrto_logits(softmax_ouput)
    D_L_by_z = util.derv_crossentropyloss_wrto_logits(softmax_ouput,target)
    # For the last layers  W= 6
    # EqA1 in https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/
    activation_L =  softmax_ouput
    DL_by_z6 = (activation_L -target)
    print("DL_by_z6.shape =", DL_by_z6.shape)
    D_L_by_w6 = DL_by_z6 *a5
    print("BP 6: Last weight update - D_L_by_w6 shape == w6 shape",D_L_by_w6.shape,w6.shape)
    w6 = w6 - lr*D_L_by_w6
    

    print("-----------------------------------")
    # For the inner layers  W= 5
    # EqA2 in https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/
    a5_derv = util.derv_sigmoid(z5)
    #DL_by_z6 = np.expand_dims(DL_by_z6, axis=1)
    DL_by_z5 = DL_by_z6 @ w6.T  # @ is np.matmul (almost similar to np.dot -https://stackoverflow.com/a/34142617/429476 )
    DL_by_z5 = DL_by_z5 * a5_derv #  Hadamard product/element wise multiplication here (20, 1) * (20, 1)
    print("DL_by_z5 shape",DL_by_z5.shape) # (20, 1)
    D_L_by_w5 =   a4.T @ DL_by_z5
    print("D_L_by_w5 shape == w5 Shape",D_L_by_w5.shape,w5.shape)
   
    # Using gradient descent to update the weights
    print("BP 5: Last weight update")
    w5 = w5 - lr*D_L_by_w5
    print("w5 shape after update", w5.shape)
   
    print("-----------------------------------")
    # For the inner layers  W= 4
    a4_derv = util.derv_sigmoid(z4)
    DL_by_z4 = DL_by_z5 @ w5.T
    DL_by_z4 = DL_by_z4 * a4_derv #  Hadamard product/element wise multiplication here (20, 120) * (20, 120)
    print("DL_by_z4 shape",DL_by_z4.shape) # (20, 120)
    print("a3 shape",a3.shape)
    #D_L_by_w4 = a3.T @ DL_by_z4 #(20,20,16).T @ (20,120) ==> (16,20,120); we need 20,120,16; need np.Transpose or below
    D_L_by_w4=np.einsum('ijp,ik->ikp', a3, DL_by_z4) 
    print("D_L_by_w4 shape == w4 Shape",D_L_by_w4.shape,w4.shape)
    # Using gradient descent to update the weights
    print("BP 4: Last weight update")
    w4 = w4 - lr*D_L_by_w4

Output

-----------Back Propogation------------------------
1
DL_by_z6.shape = (10,)
BP 6: Last weight update - D_L_by_w6 shape == w6 shape (20, 10) (20, 10)
-----------------------------------
DL_by_z5 shape (20, 20)
D_L_by_w5 shape == w5 Shape (120, 20) (120, 1)
BP 5: Last weight update
w5 shape after update (120, 20)
-----------------------------------
DL_by_z4 shape (20, 120)
a3 shape (20, 20, 16)
D_L_by_w4 shape == w4 Shape (20, 120, 16) (20, 120, 16)
BP 4: Last weight update
```
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.