1
$\begingroup$

Let's say I have the following constraints:

  1. The architecture is fixed (see image) (note that there are no biases)
  2. Activation function for the hidden layer is ReLU
  3. There's no activation function for the output layer (should just return the sum of the inputs it receive).

![enter image description here

I tried to implement this in pytorch and train to learn XOR function but I failed.

I know that a single perceptron cannot learn the xor function because it always gives a linear boundary. Here, we are summing up two perceptrons which I believe that should be able to learn the XOR. Is this understanding correct?

My other questions are:

  1. Is this a feasible problem to achieve with the given network? If yes, how?
  2. If this is doable, can we still achieve that by constraining the weights to be in the set $\{-1, 0, 1\}$
$\endgroup$
  • $\begingroup$ @asdf Can you prove your claim? $\endgroup$ – j.Doe Mar 7 '19 at 20:43
  • $\begingroup$ Sorry, my previous solution does not actually work. I'm gonna think about it later, but it looks bad at first glance, since output(T,T) = output(T,F) + output(F,T) except for the sigmoid functions, which I don't really think as playing any role here $\endgroup$ – David Mar 8 '19 at 8:09
0
$\begingroup$

Assuming your neurons have bias terms (call them $b_1$, $b_2$, and $b_3$), your solution is:

$$w_1 = 2, w_3 = -2, b_1 = -1$$

$$w_4 = -2, w_2 = 2, b_2 = -1$$

and

$$w_5 = 1, w_6 = 1, b_3 = 0$$

What is actually happening:

The first (top) hidden neuron "fires" (produces $1$ as the output) on input $[1, 0]$ and is inactive otherwise. Conversely, the other hidden neuron fires only on $[0, 1]$. In the output layer we sum these values.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.