12
$\begingroup$

What is the difference between binary encoding and one-hot for categorical input variables for English Text and their impact on the neural network? Can anyone help me to find a scientific paper about this problem?

$\endgroup$

1 Answer 1

17
$\begingroup$

If you have a system with $n$ different (ordered) states, the binary encoding of a given state is simply it's $\text{rank number} - 1$ in binary format (e.g. for the $k$th state the binary $k - 1$). The one hot encoding of this $k$th state will be a vector/series of length $n$ with a single high bit (1) at the $k$th place, and all the other bits are low (0).

As an example encodings for the next system (levels of education):

Level "Decimal encoding" Binary encoding One hot encoding
No 0 000 000001
Primary 1 001 000010
Secondary 2 010 000100
BSc/BA 3 011 001000
MSc/MA 4 100 010000
PhD 5 101 100000

References: One hot encoding at Wikipedia

And a 2017 paper on the comparison on the effects of different encodings to neural networks in the International Journal of Computer Applications could be a good starting point: A Comparative Study of Categorical Variable Encoding Techniques for Neural Network Classifiers

$\endgroup$
6
  • $\begingroup$ Thank you very much but for level of character $\endgroup$ Jan 27, 2018 at 9:30
  • 1
    $\begingroup$ What do you mean by "but for level of character"? Sorry, I don't get it. $\endgroup$
    – oszkar
    Jan 27, 2018 at 10:51
  • 1
    $\begingroup$ when make the pre-processing of data(encoding) to enter the model we represent each character to zeros and ones not word $\endgroup$ Jan 27, 2018 at 12:31
  • 1
    $\begingroup$ Also, if you have $n$ unique categories (or words here), OHE results in either $n$ or $n-1$ features where as binary encoding results in only $\log_{2} n$. So if your vocabulary is $100$ words then OHE needs at least $99$ features whereas binary encoding needs only $7$ which is a major reduction in dimensionality. $\endgroup$
    – Dan
    Feb 8, 2018 at 8:03
  • 1
    $\begingroup$ I just read this paper: diva-portal.org/smash/get/diva2:1259073/FULLTEXT01.pdf Unsuprisingly, one-hot encoded variables give the best precision. I think it's a good practice to only use binary encoding when we are limited by resources. $\endgroup$
    – neeh
    Feb 7, 2023 at 4:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.