0
$\begingroup$

I have a simple 2-2-1 fully connected network, which I suppose means it does not include bias nodes since you don't connect bias nodes at every layer. I have two sets of weights for each layer W2: (2 x 2) and W3:(1 x 2) and I only perform forward propagation using them. For backprop, I also only update these two sets of weights, like so:

 def backward(self,X,y):

    n_examples = len(X)

    yh,Z2,A2,Z3 =  self.forward(X,y)

    delta3 =(yh-y)
    de3 = np.dot(delta3,A2.T)
    delta2 = (np.dot(self.W3.T,delta3)) * self.dsigmoid(Z2)
    de2 = np.dot(delta2,X)

    self.W3 =  self.W3 - self.lr * (de3 /n_examples)
    self.W2 =  self.W2 - self.lr * (de2 /n_examples)

    return de3/n_examples ,de2 /n_examples

I have tried including a third feature ($x_{0}$ as an array of 1s) and $w_{0}$ as the corresponding bias, but this messes up the dimensions. Specifically, Z2:(2 x N) where N is the number of samples, whose dimension does not match with the dimension of np.dot(self.W3.T,delta3), which is now (3 x N) and I can no longer take their Hadamard product without appending an artificial row of 1s for the bias node (shown below in red) to $z_{2}$, to make it (3 x N), which seems like a terrible way to me. MLP with bias nodes

Is there a better way to include the bias? For instance, just adding it to $z_{3}$ during forward propagation like so:

 def forward(self,X,y):
    ...
    z3 = self.params_l2.dot(a2) + b2
    ...

But then how would I update it during backprop?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.