8
$\begingroup$

I ran into a challenge when see this solved old exam.

enter image description here

As seen in this image the author select $(D)$ as the best option with minimum node. and in another page mentioned that if we use Bipolar then $(E)$ is the answer. anyone can describe why $(E)$ is the answer when we use bipolar?

if we have step function then if input of neuron $> 0$ then output of neuron is $1$ else $0$

if we have bipolar function then if input of neuron $> 0$ then output of neuron is $1$ else $-1$

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ What does Bipolar mean here? $\endgroup$
    – gunes
    Commented Jan 6, 2021 at 22:28
  • $\begingroup$ Are there any other assumptions we need to make about the neurons? By "the input" you simply mean the sum of all inputs? And each individual input is simply the output of the previous neuron multiplied by the weight? Do the neurons have a bias term? $\endgroup$
    – Igor F.
    Commented Jan 7, 2021 at 7:54
  • $\begingroup$ Maybe one should focus on why E does not work for the binary step function. If we call n_1,n_2 the hidden neurons, what the first layer is doing is dividing the space into 4, and assigning a different tuple (0,1),(1,0), (1,1), (0,0) to (n_1,n_2) in each of the 4 regions. The last output layer should just map these tuples to the desired output. It looks that this is impossible, but I do not see at the moment any difference between the bipolar neuron and the standard one since in (n1,n_2) space the points (0,1),(1,0), (1,1), (0,0) and (-1,1),(1,-1), (1,1), (-1,-1) are arranged similarly. $\endgroup$
    – Thomas
    Commented Jan 7, 2021 at 9:20
  • $\begingroup$ I would be interested to know where is the catch :D $\endgroup$
    – Thomas
    Commented Jan 7, 2021 at 9:26
  • $\begingroup$ @kevin307505 Then I believe E is wrong. It could be right if the output neuron could multiply its inputs. Then you would have $1 \cdot 1 = 1$ and $-1 \cdot -1 = 1$ for one class, and $1 \cdot -1 = -1$ and $-1 \cdot 1 = -1$ for the other. But for additive neurons I don't see a solution. $\endgroup$
    – Igor F.
    Commented Jan 7, 2021 at 9:46

2 Answers 2

Reset to default
6
$\begingroup$

As long as we are talking only about additive neurons (i.e. all inputs to the neuron are summed together before being passed to the activation function), "unipolar" and "bipolar" can be used interchangeably. We can always transform a "unipolar" output to a "bipolar" one by multiplying by 2 and subtracting one:

$$ o_{bipolar} = 2 \cdot o_{unipolar} - 1 $$

To implement this in the network, we just need to double the weights and decrease the bias in by one for each input neuron:

$$ w_{ij}' = 2 \cdot w_{ij} $$ $$ bias_j' = bias_{j} - N_{in[j]} $$ where $N_{in[j]}$ is the number of neurons feeding their output as the input to the $j$-th neuron.

So the part

if we use Bipolar

can be safely ignored. Now, as Thomas points out in his comment, the first layer of the networks (D) and (E) simply map the continuous $(x, y)$-space onto $\{0, 1\}^2$ (or, alternatively, $\{-1, 1\}^2$, if you use "bipolar" neurons). With the given arrangement of the classes this becomes the classical XOR-problem, and you need two further layers to solve it.

Transforming to XOR

Improve this answer
$\endgroup$
3
  • $\begingroup$ (D) is right and (E) is wrong, regardless of the activation function (and I'm curious whether @Aksakal can provide a setting to prove me wrong). $\endgroup$
    – Igor F.
    Commented Jan 7, 2021 at 20:13
  • $\begingroup$ @user3661613 Please read carefully what I wrote. Not the original problem is XOR, but the output of the first layer of (E) (and of (D), too), with the optimal weightings, becomes the XOR problem. $\endgroup$
    – Igor F.
    Commented Jan 8, 2021 at 8:17
  • $\begingroup$ @kevin307505 Yes. $\endgroup$
    – Igor F.
    Commented Jan 8, 2021 at 15:48
1
$\begingroup$

If neuron had three outputs, say [-1,0,1] then it could draw three areas with linear boundaries as shown here for the first layer and solution would be (E). enter image description here The second layer simply picks the south and north region as one category, and west and east regions as another.

A neuron with two outputs, whether it's [0,1] or [-1,1] or any other pair of values, can only criss-cross. So the solution can only be (D)

Sideways

If you abstract yourself from the actual question, then it's clear that the variables are "wrong" :) This is asking for feature engineering (another buzzword!) - shift and rotate by 45 degrees would work beautifully. First you de-mean the data, then create new variables: S = x+y and V=x-y. Then your classification becomes simply a bit problem: L is (S*V<0).

No, this is not the solution of the problem, because it still requires four regions, and with binary neurons you still need D in this problem. I just thought it's an interesting twist to consider

enter image description here

Improve this answer
$\endgroup$
8
  • $\begingroup$ with what you call a step function, all you could do is criss-cross, i.e. two neurons. there's no advantage in adding more neurons in first layer. you need to multiple next. so you save your neurons for the next layer where you do it. once you multiplied, the third and final layer can pick the Ls. that's why D is the answer. $\endgroup$
    – Aksakal
    Commented Jan 7, 2021 at 15:53
  • $\begingroup$ @DaviedZuhraph yes, looks like it. the way to think of this is that the problem is to split the area into 4 parts then combine them into two relevant parts for the answer. so first layer creates south-west and north-east. then second layer first neuron creates west and the rest, while the second neuron creates east or the rest. finally the third neuron picks east and west for L, or the opposite is not L $\endgroup$
    – Aksakal
    Commented Jan 7, 2021 at 16:04
  • $\begingroup$ @Aksakal Would you mind providing the weights and the biases for the network (E) which would solve the problem? Re. Sideways: PCA is of no help here, as there is no axis along which the data vary more than along any other. What you're doing is simple rotation (a linear operation, which you can do in the network using an additional layer). Your solution, sgn(S*V), requires the output neuron to do multiplication, but the neurons in question are only allow to sum their inputs. $\endgroup$
    – Igor F.
    Commented Jan 7, 2021 at 20:10
  • $\begingroup$ @Aksakal Rotation is a part of PCA (after identifying the principal components), but not every rotation is PCA. In our case, the problem is that no component is 'principal': the dataset is, for all we can see, rotation-symmetric. So your rotation is arbitrary, not based on PCs. On a different topic: Would you mind providing the weights and biases? $\endgroup$
    – Igor F.
    Commented Jan 7, 2021 at 20:44
  • $\begingroup$ @IgorF. got your point on PCA vs rotation. I'll clarify my answer. let me think of the weights question $\endgroup$
    – Aksakal
    Commented Jan 7, 2021 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.