I have a simple 2-2-1 fully connected network, which I suppose means it does not include bias nodes since you don't connect bias nodes at every layer. I have two sets of weights for each layer W2: (2 x 2) and W3:(1 x 2) and I only perform forward propagation using them. For backprop, I also only update these two sets of weights, like so:

 def backward(self,X,y):

    n_examples = len(X)

    yh,Z2,A2,Z3 =  self.forward(X,y)

    delta3 =(yh-y)
    de3 = np.dot(delta3,A2.T)
    delta2 = (np.dot(self.W3.T,delta3)) * self.dsigmoid(Z2)
    de2 = np.dot(delta2,X)

    self.W3 =  self.W3 - self.lr * (de3 /n_examples)
    self.W2 =  self.W2 - self.lr * (de2 /n_examples)

    return de3/n_examples ,de2 /n_examples

I have tried including a third feature ($x_{0}$ as an array of 1s) and $w_{0}$ as the corresponding bias, but this messes up the dimensions. Specifically, Z2:(2 x N) where N is the number of samples, whose dimension does not match with the dimension of np.dot(self.W3.T,delta3), which is now (3 x N) and I can no longer take their Hadamard product without appending an artificial row of 1s for the bias node (shown below in red) to $z_{2}$, to make it (3 x N), which seems like a terrible way to me. MLP with bias nodes

Is there a better way to include the bias? For instance, just adding it to $z_{3}$ during forward propagation like so:

 def forward(self,X,y):
    z3 = self.params_l2.dot(a2) + b2

But then how would I update it during backprop?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.