12
$\begingroup$

Are there any good reasons for preferring binary values (0/1) over discrete or continuous normalized values, e.g. (1;3), as inputs for a feedforward network for all input nodes (with or without backpropagation)?

Of course, I'm only talking about inputs that could be transformed into either form; e.g., when you have a variable that can take several values, either directly feed them as a value of one input node, or form a binary node for each discrete value. And the assumption is that the range of possible values would be the same for all input nodes. See the pics for an example of both possibilities.

While researching on this topic, I couldn't find any cold hard facts on this; it seems to me, that - more or less - it'll always be "trial and error" in the end. Of course, binary nodes for every discrete input value mean more input layer nodes (and thus more hidden layer nodes), but would it really produce a better output classification than having the same values in one node, with a well-fitting threshold function in the hidden layer?

Would you agree that it's just "try and see", or do you have another opinion on this? Possibility one: direct input of the possible values {1;3} Possibility two: get each input value a binary node

$\endgroup$
10
$\begingroup$

Whether to convert input variables to binary depends on the input variable. You could think of neural network inputs as representing a kind of "intensity": i.e., larger values of the input variable represent greater intensity of that input variable. After all, assuming the network has only one input, a given hidden node of the network is going to learn some function $f(wx + b)$. where $f$ is the transfer function (e.g. the sigmoid) and $x$ the input variable.

This setup does not make sense for categorical variables. If categories are represented by numbers, it makes no sense to apply the function $f(wx + b)$ to them. E.g. imagine your input variable represents an animal, and sheep=1 and cow=2. It makes no sense to multiply sheep by $w$ and add $b$ to it, nor does it make sense for cow to be always greater in magnitude than sheep. In this case, you should convert the discrete encoding to a binary, 1-of-$k$ encoding.

For real-valued variables, just leave them real-valued (but normalize inputs). E.g. say you have two input variables, one the animal and one the animal's temperature. You'd convert animal to 1-of-$k$, where $k$=number of animals, and you'd leave temperature as-is.

$\endgroup$
  • $\begingroup$ So to put it short, you point to the scale of the variables: metric, ordinal and nominal. Well I think it's obvious that nominal scales can't be "calculated" or represented by a function. Regarding real values, like you I tend to think that real values might be "better" than "classified" real values due to the smoother tranisitions, but I just couldn't find any hard proof on that. Seems like another case of "trial and error" to me. $\endgroup$ – cirko Mar 25 '16 at 14:58
3
$\begingroup$

Yes there are. Imagine your goal is to build a binary classifier. Then you model your problem as estimating a Bernoulli distribution where, given a feature vector, the outcome belongs to either one class or the opposite. The output of such a neural network is the conditional probability. If greater than 0.5 you associate it to a class, otherwise to the other one.

In order to be well defined, the output must be between 0 and 1, so you choose your labels to be 0 and 1, and minimize the cross entropy, $$ E = y(x)^{t}(1-y(x))^{1-t} $$ where $y(x)$ is the output of your network, and $t$ are the target values for your training samples. Hence, you need $t \in \left\{0, 1\right\}$.

$\endgroup$
  • $\begingroup$ I understand that a normalized input is to be preferred over variable ranges of input values, as this is more similar to the binary outputs that the network shoud produce. But in my question, I wanted to refer to normalized discrete values of a certain range, i.e. if the inputs could be within a range, then all of the nodes should have the same range, i.e. be normalized. In that case, would it still be preferrable to use binary nodes for each discrete value? (I now edited the question to meet this precondition) $\endgroup$ – cirko Jun 21 '15 at 22:32
1
$\begingroup$

I also faced same dilemma when I was solving a problem. I didn't try both the architecture, but my take is, if the input variable is discrete then the output function of neural network will have the characteristic of impulse function and neural network is good at modeling impulse function. In fact any function can be modeled with neural network with varying precision depending on complexity of neural network. The only difference is, in first architecture , you have increase the number of input so you more number of weight in first hidden layer's node to model the impulse function but for the second architecture you need more number of node in hidden layer compared to first architecture to get same performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.