1
$\begingroup$

I've started out studying Machine Learning and am currently reading up about how a single perceptron works. From the wikipedia page, my understanding is as follows: suppose we have an input sample $\mathbf{x} = [x_1, \ldots, x_n]^T$, an initial weight vector $\mathbb{w} = [w_1, \ldots, w_n]^T$. Let the true output corresponding to $\mathbf{x}$ be $y'$.

The output given by the perceptron is $y = f(\sum_{i=0}^n w_ix_i)$, where $w_0$ is the bias and $x_0=1$. If $\eta$ is the learning rate, the weights are updated according to the following rule: $$\Delta w_i = \eta x_i(y'-y)$$

This is according to wikipedia. But I know the weights are updated on the basis of the gradient descent method, and I found another nice explanation based on the gradient descent method HERE. The derivation there results in the final expression for weight update:

$$\Delta w_i = \eta x_i(y'-y)\frac{df(S)}{dS}$$

where $S = \sum_{i=0}^{n}w_ix_i$. Is there a reason why this derivative term is ignored? There was another book that mentioned the same weight update formula as Wikipedia, without the derivative term. I'm pretty sure we can't just assume $f(S) = S$.

$\endgroup$
2
$\begingroup$

The difference is that the first formula is the derivation of just the output of a perceptron, while the second is the derivation of the non-linear activation of the perceptron.

When stacking perceptron layers (MLP - Multi-layered Perceptron), you have to add some non-linearity on the output of each layers, otherwise all the process is linear (and can be modeled with a single layer).

So the output of the perceptron (or more accurately, the input of the next layer) becomes:

The derivation will be as in your second formula.

If you are not using a non-linear activation (single layer), the output is:

and the derivation is as in your first formula.

$\endgroup$
  • $\begingroup$ Thanks! A couple of follow-up questions though - what's the difference between output and activation of a perceptron? As in, in the single perceptron case, won't activation and output be the same thing? Secondly, for a single perceptron, why don't we just use a non-linear output function? From what I've read, a linear output function requires that the input space is linearly separable, otherwise the weights won't converge. Won't a non-linear function be more versatile than a linear one? $\endgroup$ – Shirish Kulhari Jan 17 at 11:40
  • $\begingroup$ Also, regarding your last line, $y = S$, isn't $y$ supposed to be defined by a step function instead? In other words, $y = 1$ if $S \geq 0$ and $y = 0$ if $S < 0$. In that case, for a single perceptron, we can't even use gradient descent since $y$ is discontinuous (a step function). $\endgroup$ – Shirish Kulhari Jan 17 at 12:18
  • $\begingroup$ First comment: activation and output are not always the same, like in the linear perceptron case (where there is no activation). It depends in what resolution you are using to analyze the perceptron. Not all definitions place the activation within the perceptron block, some add it as an additional block (following right after). Eventually it is semantics. $\endgroup$ – Mark.F Jan 17 at 14:25
  • $\begingroup$ In general, people do use a non-linear output almost every single time because it offers more "versatility". But it doesn't mean that we need to ignore the linear case. $\endgroup$ – Mark.F Jan 17 at 14:27
  • $\begingroup$ Second comment: What you are referring to is the most classic case of the perceptron algorithm, which uses a step function as its non-linear activation. As you have mentioned, since it can't be used in the gradient descend training algorithm, other non-linear activations are used (for example tanh, sigmoid, ReLU, etc.). $\endgroup$ – Mark.F Jan 17 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.