8
$\begingroup$

what is the difference between binary Encoding and one-hot for categorical input variables for English Text and their impact on the neural network ? can anyone help me to find a scientific paper about this problem?

$\endgroup$
12
$\begingroup$

If you have a system with $n$ different (ordered) states, the binary encoding of a given state is simply it's $\text{rank number} - 1$ in binary format (e.g. for the $k$th state the binary $k - 1$). The one hot encoding of this $k$th state will be a vector/series of length $n$ with a single high bit (1) at the $k$th place, and all the other bits are low (0).

As an example encodings for the next system (levels of education):

-----------------------------------------------
|   Level   | "Decimal  | Binary   | One hot  |
|           | encoding" | encoding | encoding |
-----------------------------------------------
| No        |     0     |    000   |  000001  |
| Primary   |     1     |    001   |  000010  |
| Secondary |     2     |    010   |  000100  |
| BSc/BA    |     3     |    011   |  001000  |
| MSc/MA    |     4     |    100   |  010000  |
| PhD       |     5     |    101   |  100000  |
-----------------------------------------------

References: One hot encoding at Wikipedia

And a 2017 paper on the comparison on the effects of different encodings to neural networks in the International Journal of Computer Applications could be a good starting point: A Comparative Study of Categorical Variable Encoding Techniques for Neural Network Classifiers

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you very much but for level of character $\endgroup$ – عبدالله رمزي Jan 27 '18 at 9:30
  • 1
    $\begingroup$ What do you mean by "but for level of character"? Sorry, I don't get it. $\endgroup$ – oszkar Jan 27 '18 at 10:51
  • $\begingroup$ when make the pre-processing of data(encoding) to enter the model we represent each character to zeros and ones not word $\endgroup$ – عبدالله رمزي Jan 27 '18 at 12:31
  • $\begingroup$ if you find any research paper talks about this thanks in advance $\endgroup$ – عبدالله رمزي Jan 27 '18 at 12:36
  • $\begingroup$ Also, if you have $n$ unique categories (or words here), OHE results in either $n$ or $n-1$ features where as binary encoding results in only $\log_{2} n$. So if your vocabulary is $100$ words then OHE needs at least $99$ features whereas binary encoding needs only $7$ which is a major reduction in dimensionality. $\endgroup$ – Dan Feb 8 '18 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.