2
$\begingroup$

It is the case that XOR functions can be implemented using an ANN of logistic sigmoid functions.

Is this also the case using linear transfer functions? Say if the output of the overall network is decoded as a zero if less than 0.5 or one if greater or equal to 0.5.

There is no restriction on the number of nodes or layers.

I ask purely out of curiosity and couldn't find an answer via google.

(I know this decoding adds an element of non linearity but this is also how ANN of logistic sigmoid functions are decoded).

$\endgroup$
1
$\begingroup$

Yes, you can, but that would be much harder. The good choice for sigmoid is that this function is smooth alternative to step function and you can easily differentiate it in backpropagation algorithm and use it in train epoch to control your weight updates. The good reason for sigmoid that you need this function control your upper and lower bounds between 0 and 1, but linear function output defined all real numbers. So you can using, for example backpropagation, control your output, but this architecture would be much harder to learn that for sigmoid activation function, because your weights must be perfectly fitted and after dot product gives you value output near 0 and near 1. Also this almost perfect fit will probably gives your network less flexibility and it will gives you worth prediction accuracy for input values which it didn't see before.

UPDATE

Sorry, I thought about my answer and realized my mistake. So I change my answer to no. If you use linear transfer function (LTF) you will use just linear transformation, that means you will never solve it. The idea behind sigmoid function is that you make some non-linear transformations with your points and try to get some new space which would be linearly separable. But in linear operations nature you can't do the same thing. Dot product with weights just make some linear combination in your input matrix column space. I'll try to explain it in an example:

Just use the same XOR example and wrote it in matrix notation

$$X = \begin{bmatrix} 0 & 0 \\ 0 & 1 \\ 1 & 0 \\ 1 & 1 \end{bmatrix}$$ $$W_{1} = \begin{bmatrix} a & b \\ c & d \end{bmatrix}$$

where $a$, $b$, $c$ and $d$ are any real numbers. Now let's trying populating input $X$ throw the network and let check result.

$$ f(x) = x $$ $$ O = f(X \cdot W_{1}) = X \cdot W_{1} = \begin{bmatrix} 0 & 0 \\ c & d \\ a & b \\ a + c & b + d \end{bmatrix} $$

As you can see, linear transformation didn't change our space and the XOR problem is not linearly separable (just put any numbers in $a$, $b$, $c$ and $d$ variables and try visualize new points in 2D space).

So, Let me explain first layer output. First point (zero-zero) would be at the same place in space. Second and third would be some new points in space. And forth one would be a linear combination on second and third row. That mean you just build using simple parallelogram rule new vector which always would be between second and third vectors and the points is non separable. Infinity many linear layers will make the same linear changes with your input vectors.

Hope this answer helps you get better intuition in neural networks.

$\endgroup$
  • $\begingroup$ Cheers. I do know it would be a bad idea genially to do so - I was just curious. If possible you you provide an example ANN using linear transfer functions which did implement an XOR gate. No worries if not, I'm just interested to see how it would be formed. $\endgroup$ – Andy T May 13 '15 at 15:26
  • $\begingroup$ Sorry, I made huge mistake in my answer. Please check updates. $\endgroup$ – itdxer May 13 '15 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.