I am working on linear classification script that uses gradient descent to do a binary classification of an object based on two features. I'm working with just a neuron. The output of the neuron uses an activation function, in this case, the sigmoid, so that, the hypotesis value is the sum of the dot product of the weights vectors and the features vector for the i object.

$$ h = sigmoid(\sum_{i=1}^{m} \vec{x_i}·\vec{w}) $$

I have a loss function, being y the expected result, hence label for the i object

$$ L = \sum_{i=1}^{m} (h_i-y_i)^2 $$

Lastly, I have to optimize the result and upgrade the w vector so that the loss function reaches minimum. For that I need to calculate the gradient. Being i the object and j the column of the features vector.

$$ \vec{gradient(j)} = \sum_{i=1}^{m} 2*(h_i-y_i) · {x_{ij}} $$

Lastly, the weights vector is updated:

$$ \vec{w(i+1)} = \vec{w(i)} - learningRate · \vec{gradient} $$

Now, my question is, should sigmoid be applied in the gradient calculation ? I'm getting confused with the loss function results as they all head straight to 0 after the first epoch at no matter what learning rate. The accuracy is great thought but I'm expecting some divergence on big learning rates and the loss function just gets stuck at the minimum.

In short, I'd like to know when I need to use the sigmoid function.

Very appreciated


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy