Let's say I have the following constraints:

  1. The architecture is fixed (see image) (note that there are no biases)
  2. Activation function for the hidden layer is ReLU
  3. There's no activation function for the output layer (should just return the sum of the inputs it receive).

![enter image description here

I tried to implement this in pytorch and train to learn XOR function but I failed.

I know that a single perceptron cannot learn the xor function because it always gives a linear boundary. Here, we are summing up two perceptrons which I believe that should be able to learn the XOR. Is this understanding correct?

My other questions are:

  1. Is this a feasible problem to achieve with the given network? If yes, how?
  2. If this is doable, can we still achieve that by constraining the weights to be in the set $\{-1, 0, 1\}$
  • $\begingroup$ @asdf Can you prove your claim? $\endgroup$
    – j.Doe
    Mar 7, 2019 at 20:43
  • $\begingroup$ Sorry, my previous solution does not actually work. I'm gonna think about it later, but it looks bad at first glance, since output(T,T) = output(T,F) + output(F,T) except for the sigmoid functions, which I don't really think as playing any role here $\endgroup$
    – David
    Mar 8, 2019 at 8:09

1 Answer 1


Assuming your neurons have bias terms (call them $b_1$, $b_2$, and $b_3$), your solution is:

$$w_1 = 2, w_3 = -2, b_1 = -1$$

$$w_4 = -2, w_2 = 2, b_2 = -1$$


$$w_5 = 1, w_6 = 1, b_3 = 0$$

What is actually happening:

The first (top) hidden neuron "fires" (produces $1$ as the output) on input $[1, 0]$ and is inactive otherwise. Conversely, the other hidden neuron fires only on $[0, 1]$. In the output layer we sum these values.

  • $\begingroup$ OP specified no bias terms, but it's not hard to modify these weights to work without them: (1, -1) works fine (and so the answer to extra question 2 is "yes"). $\endgroup$ Nov 4, 2021 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.