This is a rather long question. Sorry for that. The main thing is that I tried to derive out backpropagation via chain rule etc. I am aware that the index notation changes to transpose the matrix while doing multiplication; and also that for matrix calculus Hadamard product is needed. However, I am not exactly clear where each has to apply. I derived out the equation of the last layer and inner layers with index notation. I tried to implement a simple CNN to test this out; however, during backpropagation, the weights are not matching. Even when I try to do Hadamard product (element-wise multiplication). Not able to break through.
Excerpt from https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/
Neural Network
$$ \mathbf { \bbox[10px, border:2px solid red] { \color{red}{ \begin{aligned} a^0 \rightarrow \bbox[5px, border:2px solid black] { \underbrace{\text{hidden layers}}_{a^{l-2}} } \,\rightarrow \bbox[5px, border:2px solid black] { \underbrace{w^{l-1} a^{l-2}+b^{l-1}}_{z^{l-1} } } \,\rightarrow \bbox[5px, border:2px solid black] { \underbrace{\sigma(z^{l-1})}_{a^{l-1}} } \,\rightarrow \bbox[5px, border:2px solid black] { \underbrace{w^l a^{l-1}+b^l}_{z^{l}/logits } } \,\rightarrow \bbox[5px, border:2px solid black] { \underbrace{P(z^l)}_{\vec P/ \text{softmax} /a^{l}} } \,\rightarrow \bbox[5px, border:2px solid black] { \underbrace{L ( \vec P, \vec Y)}_{\text{CrossEntropyLoss}} } \end{aligned} }}} $$
CrossEntropy Loss with respect to Weight in last layer
$$ \mathbf { \frac {\partial L}{\partial w^l} = \color{red}{\frac {\partial L}{\partial z^l}}.\color{green}{\frac {\partial z^l}{\partial w^l}} \rightarrow \quad EqA1 } $$ $$ \color{red} { \begin{aligned} \frac {\partial L}{\partial z^l} = p_i - y_i \rightarrow \quad \text{EqA1.1} \end{aligned} } $$ We need to put this back in $EqA1$. We now need to calculate the second term, to complete the equation
$$ \begin{aligned} \frac {\partial L}{\partial w^l} = \color{red}{\frac {\partial L}{\partial z^l}}.\color{green}{\frac {\partial z^l}{\partial w^l}} \\ \\ z^{l} = (w^l a^{l-1}+b^l) \\ \color{green}{\frac {\partial z^l}{\partial w^l} = a^{l-1}} \\ \\ \text{Putting all together} \\ \\ \frac {\partial L}{\partial w^l} = (p_i - y_i) *a^{l-1} \quad \rightarrow \quad \mathbf {EqA1} \end{aligned} $$
Using Gradient descent we can keep adjusting the last layer like
$$ w{^l}{_i} = w{^l}{_i} -\alpha * \frac {\partial L}{\partial w^l} $$
Now let's do the derivation for the inner layers
Derivative of Loss with respect to Weight in Inner Layers
he trick here (yes it is a trick), is to derivative the Loss with respect to the inner layer as a composition of the partial derivative we computed earlier. And also to compose each partial derivative as partial derivative with respect to either $z^x$ or $w^x$ but not with respect to $a^x$. This is to make derivatives easier and intuitive to compute.
$$ \begin{aligned} \frac {\partial L}{\partial w^{l-1}} = \color{blue}{\frac {\partial L}{\partial z^{l-1}}}. \color{green}{\frac {\partial z^{l-1}}{\partial w^{l-1}}} \rightarrow \text{EqA2} \end{aligned} $$
The trick is to represent the first part in terms of what we computed earlier; in terms of $\color{blue}{\frac {\partial L}{\partial z^{l}}}$
$$ \begin{aligned} \color{blue}{\frac {\partial L}{\partial z^{l-1}}} = \color{blue}{\frac {\partial L}{\partial z^{l}}}. \frac {\partial z^{l}}{\partial a^{l-1}}. \frac {\partial a^{l-1}}{\partial z^{l-1}} \rightarrow \text{ Eq with respect to Prev Layer} \\ \\ \color{blue}{\frac {\partial L}{\partial z^{l}}} = \color{blue}{(p_i- y_i)} \text{ from the previous layer (from EqA1.1) } \\ \\ z^l = w^l a^{l-1}+b^l \text{ which makes } {\frac {\partial z^{l} }{\partial a^{l-1}} = w^l} \\ \text{ and } a^{l-1} = \sigma (z^{l-1}) \text{ which makes } \frac {\partial a^{l-1}}{\partial z^{l-1}} = \sigma \color{red}{'} (z^{l-1} ) \\ \\ \text{ Putting together we get the first part of Eq A2 } \\\\ \color{blue}{\frac {\partial L}{\partial z^{l-1}}} =\color{blue}{(p_i- y_i)}.w^l.\sigma \color{red}{'} (z^{l-1} ) \rightarrow \text{EqA2.1 } \\ \\ z^{l-1} = w^{l-1} a^{l-2}+b^{l-1} \text{ which makes } \color{green}{\frac {\partial z^{l-1}}{\partial w^{l-1}}=a^{l-2}} \\ \\ \frac {\partial L}{\partial w^{l-1}} = \color{blue}{\frac {\partial L}{\partial z^{l-1}}}. \color{green}{\frac {\partial z^{l-1}}{\partial w^{l-1}}} = \color{blue}{(p_i- y_i)}.w^l.\sigma \color{red}{'} (z^{l-1} ).\color{green}{a^{l-2}} \end{aligned} $$
So far so good; Now here is the code snippet where I try to apply this. For the last layer it works. But for the inner layer it does not match weights. Which means I am somewhere wrong
Excerpt from https://github.com/alexcpn/cnn_in_python
if __name__ == '__main__':
np.set_printoptions(formatter={'float': lambda x: "{0:0.2f}".format(x)})
# Generate a random image
image_size = 32
image_depth = 3
image = np.random.rand(image_size, image_size)
# to mimic RGB channel
image = np.stack([image,image,image], axis=image_depth-1) # 0 to 2
print("Image Shape=",image.shape)
print("Image [0,0,:]=",image[0,1,2])
# The class containing the convolution Logic
testConv2D = cnn.Conv2D()
# we will create leNet without the Pooling parts
# (stride is always 1 for now)
# [32.32.3] *(5.5.3)*6 == [28.28.6] * (5.5.6)*1 = [24.24.1] * (5.5.3)*16 = [20.20.16] * FC layer
print("-----------Forward Propagation------------------------")
# For layer 1
filter_size = 5
number_of_filters = 6
# Initialize the weight's/filters of Layer1
w1 = initializeWeights(number_of_filters,filter_size,image.shape[2])
a1 = layerConvolutionActivation(image, filter_size,number_of_filters,w1)
# For layer 2
filter_size = 5
number_of_filters = 1
# Initialize the weight's/filters of Layer1
w2 = initializeWeights(number_of_filters,filter_size,a1.shape[2])
# Do convolution and activation
a2 = layerConvolutionActivation(a1, filter_size,number_of_filters,w2)
# For layer 3
# Out=W−F+1 imagesize - filtersize + 1
filter_size = 5
number_of_filters = 16
# Initialize the weight's/filters of Layer1
w3 = initializeWeights(number_of_filters,filter_size,a2.shape[2])
# Do convolution and activation
a3 = layerConvolutionActivation(a2, filter_size,number_of_filters,w3)
print("a3.shape=", a3.shape) # output_layer3 shape = (20, 20, 16)
"""
Lets add the fully connected layer say 120 - we need the shape to be compatible - for that we are adding the
the dimension of the above layer
"""
w4 = np.random.rand(a3.shape[0],120,a3.shape[2])
print("w4.shape =", w4.shape) # (20, 120, 16)
# this time there is no convolution - rather we need to do a dot
z4 = np.einsum('ijp,jkp->ik', a3, w4) # (20, 120)
#output_layer4 = np.tensordot(output_layer3,weight_layer4,axes=2)
print("z4.shape =", z4.shape)
a4 = util.sigmoid(z4)
print("a4.shape =", a4.shape)
w5 = np.random.rand(a4.shape[1],1)
print("w5.shape =", w5.shape) # (20, 120, 16)
# this time there is no convolution - rather we need to do a dot
z5 = np.einsum('ij,jk->ik', a4, w5) # (20, 120)
print("z5.shape =", z5.shape)
a5 = util.sigmoid(z5)
print("a5.shape =", a5.shape)
a5_derv = util.derv_sigmoid(z5)
print("a5_derv.shape =", a5_derv.shape)
# final layer lets make it 10 classes
w6 = np.random.rand(a5.shape[0],10)
print("w6.shape =", w6.shape) # (20, 120, 16)
# this time there is no convolution - rather we need to do a dot
logits = np.einsum('ij,ik->jk', a5, w6).flatten() # 20,1*20,10 (1, 10) == Z_l
z6 = logits
print("z6.shape=", z6.shape) #(10,)
#print("Final Output =", output_layer6)
"""
Run Softmax
"""
softmax_ouput =util.softmax(logits)
a6 = softmax_ouput
print("a6.shape =", a6.shape)
print("Softmax Output =", softmax_ouput)
# Assume that the truth was class 1 , for this particular "image"
target = np.array([1., 0., 0., 0., 0. ,0., 0. ,0 ,0., 0.])
#Plug this into the cost function lets take the CrossEntropy Loss as this a classification
# See this https://www.youtube.com/watch?v=dEXPMQXoiLc
# Get index of the true class
LcrossEntropyLoss = util.crossentropyloss(softmax_ouput,target)
print("crossEntropyLoss = ",LcrossEntropyLoss) # ex 3.75671634607845
"""
Back-Propagate the Loss
"""
print("-----------Back Propagation------------------------")
lr = 1 # learning rate
# https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/
D_S_by_z = util.derv_softmax_wrto_logits(softmax_ouput)
D_L_by_z = util.derv_crossentropyloss_wrto_logits(softmax_ouput,target)
# For the last layers W= 6
# EqA1 in https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/
activation_L = softmax_ouput
B0 = (activation_L -target)
print("B0.shape =", B0.shape)
D_L_by_w6 = B0 *a5
print("BP: Last weight update - D_L_by_w6 shape==w6 shape",D_L_by_w6.shape,w6.shape)
w6 = w6 - lr*D_L_by_w6
print("-----------------------------------")
# For the inner layers W= 5
# EqA2 in https://alexcpn.github.io/html/NN/ml/7_backpropogation_full/
B1 = w6 * B0
B1 = B1 @ a5_derv
print("B1 shape",B1.shape)
D_L_by_w5 = a4.T @ B1
print("D_L_by_w5 shape",D_L_by_w5.shape)
D_L_by_w5 = np.expand_dims(D_L_by_w5, axis=1)
print("D_I_by_w5 shape",D_L_by_w5.shape)
print("w5 shape", w5.shape)
w5 = w5 - lr*D_L_by_w5
print("w5 shape", w5.shape)
Output
cnn_1/cnn_py$ python3 main.py
Image Shape= (32, 32, 3)
Image [0,0,:]= 0.5815285626863921
-----------Forward Propogation------------------------
Convolution Shape = (28, 28, 6)
Convolution Shape = (24, 24, 1)
Convolution Shape = (20, 20, 16)
a3.shape= (20, 20, 16)
w4.shape = (20, 120, 16)
z4.shape = (20, 120)
a4.shape = (20, 120)
w5.shape = (120, 1)
z5.shape = (20, 1)
a5.shape = (20, 1)
a5_derv.shape = (20, 1)
w6.shape = (20, 10)
z6.shape= (10,)
a6.shape = (10,)
Softmax Output = [0.02 0.26 0.03 0.08 0.09 0.04 0.07 0.01 0.01 0.39]
crossEntropyLoss = 4.091121759807155
-----------Back Propogation------------------------
1
B0.shape = (10,)
BP: Last weight update - D_L_by_w6 shape==w6 shape (20, 10) (20, 10)
-----------------------------------
Traceback (most recent call last):
File "/home/alex/coding/cnn_1/cnn_py/main.py", line 172, in <module>
B1 = B1 @ a5_derv
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 20 is different from 10)
