0
$\begingroup$

I'm trying to train single layer perceptron. I don't understand one thing. How to calculate output for vector input x?

For example let say we have this data to use for training:

L1 = [1.5 2.4; -0.3 3; 2 3; 2 2.5];
L2 = [2.5 1.4; -1.3 2; 3 3.1; 1.2 2.5];

weight init, step:

step = 0.1
w = [1 1];

Then L = [L1; L2]; Then we have this formula to calculate mean sum square sum_all - I mean sigma symbol which means sum all elements.

E = 1/N * sum_all(d(j) - o(j))^2

j - is index of vector inside L matrix.

Then new weight equation:

w(new) = w + step/N * sum_all(d(j) - o(j) * o(j) * (1 - o(j)) * x(j))

Then we have this logical function (pseudo code as it may look clearer):

d(j) = 1 if x(j) belongs to L1
d(j) = 0 if x(j) belongs to L2

And I don't understand how I have to get o(j) or in other words output. I didn't find anywhere where it would say how you get it. It kind of not clear for me.

Should I do something like this?

if w * x(j)' > 0 then o(j) = 1
elseif w * x(j)' <= then o(j) = 0

But then if I run my algorithm, it stops training after one epoch and of course does not train it at all (as actual output won't differ from desired output in a first epoch at all).

If I try something like this:

o(j) = w * x(j)', then numbers go so high that matlab stop recognizing it as number and it goes into infinite loop..

So how I should get actual output x(j) - o(j) using single layer perceptron and backpropagation algorithm?..

$\endgroup$
1
$\begingroup$

Based on your update pseudo code it looks like you're using a sigmoid output. In this case given an input $x$ your output should be $$ \sigma(x) = \frac{1}{1 + e^{-(w^T x + b)}}. $$ It is worth noting that if you assume a log-loss function (which is what you should use for classification) your setup is just binary logistic regression. In this case your update should be $$ w_\text{new} = w_\text{old} + \eta \sum_{j}[d(j) - \sigma(x(j))] x(j), $$ where $\eta$ is the learning rate. If on the other hand you're using a squared loss (which it seems like you are, but is normally not the right choice for classification) the update will look like $$ w_\text{new} = w_\text{old} + \eta\sum_{j}[d(j) - \sigma(x(j))][1 - \sigma(x(j))]\sigma(x(j))x(j). $$ The reason your weights are diverging is probably due to forgetting the parenthesis when translating the above, you have

w(new) = w + step/N * sum_all(d(j) - o(j) * o(j) * (1 - o(j)) * x(j))

which should be

w(new) = w + step/N * sum_all((d(j) - o(j)) * o(j) * (1 - o(j)) * x(j))

All that being said the above isn't really the standard perceptron algorithm. Normally the output for a perceptron is given by

$$ f(x) = \mathbb{I}\{w^T x > 0\}, $$

where $\mathbb{I}$ is the indicator function. In this case you can learn the parameters using subgradient descent which results in an update of the form

$$ w_\text{new} = \left\{\begin{array}{l l} w_\text{old} + \eta [d(j) - f(x(j))] x(j) &:\; d(j) \neq f(x(j)) \\ w_\text{old} &:\; \text{otherwise.} \end{array}\right. $$

$\endgroup$
  • $\begingroup$ Thnanks. Output equation was what I was looking for. $\endgroup$ – Andrius Nov 18 '13 at 17:15
1
$\begingroup$

"I don't understand one thing. How to calculate output for vector input x?"

If by single layer perceptron you mean the input layer plus the output layer:

Zero Hidden Layers

Then for each input to the output node, take the values applied to the inputs and multiply them by their cosponsoring weight values. Then sum these weighted inputs. This sum of weighted inputs is then passed into the activation function you are using (linear summation, sigmoidal, gaussian etc) to produce the output o for those inputs.

"If I try something like this:

o(j) = w * x(j)', then numbers go so high that matlab stop recognizing it as number and it goes into infinite loop."

As far as I recall if left unguarded back propagation can push the weight values to become larger and larger. A possible solution is to cap them at a certain value.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.