I've been training a fully connected neural network I've developed so that it can learn the XOR problem. I got succesful results using hyperbolic tangent and ReLU as activation functions, this is, the network output matched with the outputs of the XOR truth table. Still, as far as I understand, the activation function should be chosen taking into account the input range, which in this case is $[0, 1]$. As that range is the active input range of the logistic function I wanted to use the latter as activation function.

Using the logistic function I get completely random results, as thay are close to $0.5$ in all cases, i.e. any input combination of $0$'s and $1$'s results in a value close to $0.5$. This leads me to think that the each output is just a guess.

What I don't understand is why if my input is bounded in the $[0, 1]$ range it works with activation functions with output range of $(-1, 1)$ or $[0, +inf)$ and not $(0, 1)$? Does my reasoning make sense or am I missing something?

Thanks in advanced.

EDIT: I've tested another set of outputs for the same group of inputs, more specifically inputs = [[0, 0], [0, 1], [1, 0], [1, 1]] and outputs = [[1, 1], [1, 0], [0, 1], [0, 0]], and get correct results: [0, 0] --> [ 0.99999543, 0.99488362] [0, 1] --> [ 9.67808797e-01, 4.01490200e-04] [1, 0] --> [ 3.19309525e-05, 9.92688220e-01] [1, 1] --> [ 0.0216361, 0.0097268] Other cases with 2 or more outputs work as well, but I still can make the XOR problem, with one output, work. Why would the network, using logistic functions as activations, classify samples correctly when having 2 or more outputs and not when there is only one?

  • $\begingroup$ According to this, the activation function should be sigmoidal. I understand this as inputs of [0,+inf] to yield [0,1] in the (y-axis) output. $\endgroup$ – Carl Jan 14 '17 at 13:22
  • $\begingroup$ @Carl, yes I agree, but what happens with ReLUs? They still can be used as activation functions, can't they? I've edited the post, the last paragraph, I mixed up. My bad. $\endgroup$ – tulians Jan 14 '17 at 13:41
  • $\begingroup$ ReLUs is just a rectifier, AKA, positive ramp; max[0,x], for which input of [-inf,+inf] yields an output of [0,+inf]. Why useful, don't know, not my area. $\endgroup$ – Carl Jan 14 '17 at 13:56
  • $\begingroup$ @Carl, thanks for pointing that out. Still, when I use the tanh, which is a sigmoid, the NN correctly classifies new data. I don't know why the logistic function does not work in this case, as the output I'm looking for is the same as the output the logistic provides. $\endgroup$ – tulians Jan 14 '17 at 14:02
  • $\begingroup$ Logistic-1/2 is sigmoidal, logistic is not. Rectifiers function used to convert alternating current to direct current and 4 rectifiers, suitably arranged, are the absolute value function; |x|. Logistic is derivative of softplus, where softplus is smooth approximation to rectifier. $\endgroup$ – Carl Jan 14 '17 at 14:39

Long story short: $\text{Logistic}(0)=0.5$, $\text{Sigmoidal(0)}=0$. Also note, $2\text{Logistic}(0)-1=0$, and one can combine a pair of logistic functions to obtain $f(0)=0$.

| cite | improve this answer | |
  • 1
    $\begingroup$ I've made the necessary changes to meet what you state here and in previous comments. Now it works correctly. Thanks for your help. $\endgroup$ – tulians Jan 15 '17 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.