0
$\begingroup$

I've been training a fully connected neural network I've developed so that it can learn the XOR problem. I got succesful results using hyperbolic tangent and ReLU as activation functions, this is, the network output matched with the outputs of the XOR truth table. Still, as far as I understand, the activation function should be chosen taking into account the input range, which in this case is $[0, 1]$. As that range is the active input range of the logistic function I wanted to use the latter as activation function.

Using the logistic function I get completely random results, as thay are close to $0.5$ in all cases, i.e. any input combination of $0$'s and $1$'s results in a value close to $0.5$. This leads me to think that the each output is just a guess.

What I don't understand is why if my input is bounded in the $[0, 1]$ range it works with activation functions with output range of $(-1, 1)$ or $[0, +inf)$ and not $(0, 1)$? Does my reasoning make sense or am I missing something?

Thanks in advanced.

EDIT: I've tested another set of outputs for the same group of inputs, more specifically inputs = [[0, 0], [0, 1], [1, 0], [1, 1]] and outputs = [[1, 1], [1, 0], [0, 1], [0, 0]], and get correct results: [0, 0] --> [ 0.99999543, 0.99488362] [0, 1] --> [ 9.67808797e-01, 4.01490200e-04] [1, 0] --> [ 3.19309525e-05, 9.92688220e-01] [1, 1] --> [ 0.0216361, 0.0097268] Other cases with 2 or more outputs work as well, but I still can make the XOR problem, with one output, work. Why would the network, using logistic functions as activations, classify samples correctly when having 2 or more outputs and not when there is only one?

Improve this question
$\endgroup$
8
  • $\begingroup$ According to this, the activation function should be sigmoidal. I understand this as inputs of [0,+inf] to yield [0,1] in the (y-axis) output. $\endgroup$
    – Carl
    Commented Jan 14, 2017 at 13:22
  • $\begingroup$ @Carl, yes I agree, but what happens with ReLUs? They still can be used as activation functions, can't they? I've edited the post, the last paragraph, I mixed up. My bad. $\endgroup$
    – tulians
    Commented Jan 14, 2017 at 13:41
  • $\begingroup$ ReLUs is just a rectifier, AKA, positive ramp; max[0,x], for which input of [-inf,+inf] yields an output of [0,+inf]. Why useful, don't know, not my area. $\endgroup$
    – Carl
    Commented Jan 14, 2017 at 13:56
  • $\begingroup$ @Carl, thanks for pointing that out. Still, when I use the tanh, which is a sigmoid, the NN correctly classifies new data. I don't know why the logistic function does not work in this case, as the output I'm looking for is the same as the output the logistic provides. $\endgroup$
    – tulians
    Commented Jan 14, 2017 at 14:02
  • $\begingroup$ Logistic-1/2 is sigmoidal, logistic is not. Rectifiers function used to convert alternating current to direct current and 4 rectifiers, suitably arranged, are the absolute value function; |x|. Logistic is derivative of softplus, where softplus is smooth approximation to rectifier. $\endgroup$
    – Carl
    Commented Jan 14, 2017 at 14:39

1 Answer 1

Reset to default
0
$\begingroup$

Long story short: $\text{Logistic}(0)=0.5$, $\text{Sigmoidal(0)}=0$. Also note, $2\text{Logistic}(0)-1=0$, and one can combine a pair of logistic functions to obtain $f(0)=0$.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I've made the necessary changes to meet what you state here and in previous comments. Now it works correctly. Thanks for your help. $\endgroup$
    – tulians
    Commented Jan 15, 2017 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.