Can we use perceptron training algorithm to train a single neuron with sigmoid activation?

Question

The perceptron training algorithm is summarized as:

• Apply the inputs and calculate the output $ y $
• Compare with the desired output yd and calculate error $e = y-y_d$
• Update the weights based on the error: $w_t = w_{t-1} + \eta ex$

I studied this for perceptron with step activation and found many examples of the training process for that like AND gate and OR gate.

The question now is: can we use this to train a single neuron with sigmoid activation? Or we must use the gradient descent.

I searched many times and found no answer to this.

I think that the samples that should be classified as one of the two classes (output = 0 or 1) will be trained successfully. However, I'm not sure about the samples that lie between the two classes (0 < output < 1). I am not sure about this and I have no proof.

Answer 1

Assume you use a sigmoid neuron with binary cross entropy loss: $$H(y)=y_d\log y+(1-y_d)\log(1-y)$$ where $y=\sigma(w^Tx)$. The gradient wrt $w$ would be: $$\begin{align}\frac{\partial H}{\partial w}&=\frac{\partial H}{\partial y}\frac{\partial y}{\partial w}=\left(\frac{y_d}{y}-\frac{1-y_d}{1-y}\right)y(1-y)x\\&=(y_d(1-y)-(1-y_d)y)x\\&=(y_d-y)x\end{align}$$ When we substitute into the gradient descent rule, we have the following update equation: $$w_t=w_{t-1}-\eta {\partial H \over {\partial w}}=w_{t-1}+\eta(y-y_d)x$$

which is the update rule you have for the Perceptron algorithm. In other words, if you use the Perceptron algorithm together with sigmoid activation, instead of heaviside step, you end up using gradient descent algorithm on binary cross-entropy loss function.