0
$\begingroup$

I'm implementing a Single Layer Perceptron for binary classification in python. I'm using binary Cross-Entropy loss function and gradient descent.

The gradient descent is not converging, may be I'm doing it wrong. Here what I did:

We have a vector of weights $W = \begin{pmatrix} w_1, \dots, w_M \end{pmatrix}$, a matrix of $N$ samples $X = \begin{bmatrix} x_{11} & \dots & x_{N,1} \\ \vdots & \ddots & \vdots\\ x_{1M} & \dots & x_{NM} \end{bmatrix}$, with each column representing a sample, a sigmoid $\sigma(x) = \frac{1}{1 + e^{-x}}$ as activation function and vector of $N$ targets $Y = \begin{pmatrix} y_1, \dots, y_N \end{pmatrix}$.

On forward step we have $V = WX$ and the output is $Z = \sigma(V)$.

On backpropagation we update the weights as $W(n+1) = W(n) - \eta \frac{\partial L}{\partial W}$, where $L$ is the binary Cross-Entropy loss function: $L(Y, Z)-\frac{1}{N}\sum_{k = 1}^{N} y_k \log(z_k) + (1 - y_k) \log(1 - z_k)$. In matrix notation this function can be rewritten as $L(Y,Z) = -\frac{1}{N} \left (Y (\log(Z))^T + (1_N - Y) (\log(1_N - Z))^T \right )$.

I think that all is correct so far and may be I messed up on the derivates.

Applying chain rule: $\frac{\partial L}{\partial W} = \frac{\partial L}{\partial Z} \frac{\partial Z}{\partial V} \frac{\partial V}{\partial W}$

$\frac{\partial V}{\partial W}$ is straightforward: $\frac{\partial V}{\partial W} = X$.

$\frac{\partial Z}{\partial V} = \sigma '(V)$

$\frac{\partial L}{\partial Z} = -\frac{1}{N} \left (Y\begin{pmatrix}\frac{1}{z_1}, \dots, \frac{1}{z_N}\end{pmatrix}^T + (1_N - Y)\begin{pmatrix}\frac{1}{z_1 - 1}, \dots, \frac{1}{z_N - 1}\end{pmatrix}^T \right)$

So finally, $W(n+1) = W(n) - \eta \left( -\frac{1}{N} \left (Y\begin{pmatrix}\frac{1}{z_1}, \dots, \frac{1}{z_N}\end{pmatrix}^T + (1_N - Y)\begin{pmatrix}\frac{1}{z_1 - 1}, \dots, \frac{1}{z_N - 1}\end{pmatrix} ^T \right) \sigma '(V) X \right)$

Are those derivates correct?

Here's my code:

# Activation function
def sigmoid(self, x):
    return 1 / (1 + np.exp(-x))

# Activation function derivative
def d_sigmoid(self, x): # derivate of sigmoid
    return self.sigmoid(x)*(1 - self.sigmoid(x))

# Binary Cross-Entropy loss function
def loss(self, Z, Y):
    Z = Z.flatten() # reshape matrix to vector
    return np.asscalar((-1/len(Z)) * (np.dot(Y, np.log(Z + (1.e-10))) + np.dot((1 - Y), np.log(1 - Z + (1.e-10))))) 

# Gradient of loss function
def gradient_loss(self, Z, Y):
    Z = Z.flatten() # reshape matrix to vector
    dL = np.dot(Y, 1/Z) + np.dot((1 - Y), -1/(1 - Z))
    return np.asscalar((-1/len(Z)) * dL)

# Foward step
def forward(self, X):
    V = self.W @ X
    Z = self.sigmoid(V)
    print("Z {}:".format(Z))
    return V,Z

# Train neural network
def train(self, X, Y):
    n = 0
    print("Epoch {}:".format(n))
    V, Z = self.forward(X)
    print("loss: {}".format(self.loss(Z,Y)))
    # Perform a gradient descent algorithm
    while self.loss(Z, Y) > 0.1 and n < 10000:
        n = n + 1
        W_new = self.W - self.rate * self.gradient_loss(Z, Y) * self.d_sigmoid(V) @ X.transpose()
        self.W = W_new
        print(self.W.shape)

        print("Epoch {}:".format(n))
        V, Z = self.forward(X)
        print("loss: {}".format(self.loss(Z,Y)))
$\endgroup$
  • $\begingroup$ The last expression you wrote has dimension mismatches. It's best if you could write your derivate for one sample and for just one $w_j$; then we can generalize it together? $\endgroup$ – gunes Sep 11 '19 at 18:23
  • $\begingroup$ Yes, on my code I used $X^T$ on the last expression. I'm gonna try derivate for one sample and one weight to check if there is any error $\endgroup$ – Henrique Andrade Sep 11 '19 at 18:30
1
$\begingroup$

You've some error in cross-entropy differentiation, thinking with $N=1$ for simplicity it should be: $$\frac{\partial L}{\partial z}=-y\frac{1}{z}-(1-y)\frac{1}{z-1}$$ But, in your formulation there is $1$ (i.e. $1_N$) before $\frac{1}{z-1}$, which doesn't belong there. Also, just be careful about sigmoid derivative, you didn't write it explicitly: $\sigma'(v)=\sigma(v)(1-\sigma(v))$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you, but actually I had derivated as you did and I've made this mistake when I transcripted the expression to the question. I'm gonna edit it. The expression that I used on my code is that one you wrote and it's not working. And we still this problem with the dimension mismatches. $\endgroup$ – Henrique Andrade Sep 11 '19 at 18:56
  • $\begingroup$ Can you share your code as well? We'll deal with dimension mismatch later, since the gradient will be a sum over the samples. $\endgroup$ – gunes Sep 11 '19 at 19:00
  • $\begingroup$ I've included it on the question $\endgroup$ – Henrique Andrade Sep 11 '19 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.