# Neural Network misclassification using logistic function

I've been training a fully connected neural network I've developed so that it can learn the XOR problem. I got succesful results using hyperbolic tangent and ReLU as activation functions, this is, the network output matched with the outputs of the XOR truth table. Still, as far as I understand, the activation function should be chosen taking into account the input range, which in this case is $[0, 1]$. As that range is the active input range of the logistic function I wanted to use the latter as activation function.

Using the logistic function I get completely random results, as thay are close to $0.5$ in all cases, i.e. any input combination of $0$'s and $1$'s results in a value close to $0.5$. This leads me to think that the each output is just a guess.

What I don't understand is why if my input is bounded in the $[0, 1]$ range it works with activation functions with output range of $(-1, 1)$ or $[0, +inf)$ and not $(0, 1)$? Does my reasoning make sense or am I missing something?

EDIT: I've tested another set of outputs for the same group of inputs, more specifically inputs = [[0, 0], [0, 1], [1, 0], [1, 1]] and outputs = [[1, 1], [1, 0], [0, 1], [0, 0]], and get correct results:  [0, 0] --> [ 0.99999543, 0.99488362] [0, 1] --> [ 9.67808797e-01, 4.01490200e-04] [1, 0] --> [ 3.19309525e-05, 9.92688220e-01] [1, 1] --> [ 0.0216361, 0.0097268]  Other cases with 2 or more outputs work as well, but I still can make the XOR problem, with one output, work. Why would the network, using logistic functions as activations, classify samples correctly when having 2 or more outputs and not when there is only one?

• According to this, the activation function should be sigmoidal. I understand this as inputs of [0,+inf] to yield [0,1] in the (y-axis) output. – Carl Jan 14 '17 at 13:22
• @Carl, yes I agree, but what happens with ReLUs? They still can be used as activation functions, can't they? I've edited the post, the last paragraph, I mixed up. My bad. – tulians Jan 14 '17 at 13:41
• ReLUs is just a rectifier, AKA, positive ramp; max[0,x], for which input of [-inf,+inf] yields an output of [0,+inf]. Why useful, don't know, not my area. – Carl Jan 14 '17 at 13:56
• @Carl, thanks for pointing that out. Still, when I use the tanh, which is a sigmoid, the NN correctly classifies new data. I don't know why the logistic function does not work in this case, as the output I'm looking for is the same as the output the logistic provides. – tulians Jan 14 '17 at 14:02
• Logistic-1/2 is sigmoidal, logistic is not. Rectifiers function used to convert alternating current to direct current and 4 rectifiers, suitably arranged, are the absolute value function; |x|. Logistic is derivative of softplus, where softplus is smooth approximation to rectifier. – Carl Jan 14 '17 at 14:39

Long story short: $\text{Logistic}(0)=0.5$, $\text{Sigmoidal(0)}=0$. Also note, $2\text{Logistic}(0)-1=0$, and one can combine a pair of logistic functions to obtain $f(0)=0$.