What is the difference between binary encoding and one-hot for categorical input variables for English Text and their impact on the neural network? Can anyone help me to find a scientific paper about this problem?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
If you have a system with $n$ different (ordered) states, the binary encoding of a given state is simply it's $\text{rank number} - 1$ in binary format (e.g. for the $k$th state the binary $k - 1$). The one hot encoding of this $k$th state will be a vector/series of length $n$ with a single high bit (1) at the $k$th place, and all the other bits are low (0).
As an example encodings for the next system (levels of education):
| Level | "Decimal encoding" | Binary encoding | One hot encoding |
|---|---|---|---|
| No | 0 | 000 | 000001 |
| Primary | 1 | 001 | 000010 |
| Secondary | 2 | 010 | 000100 |
| BSc/BA | 3 | 011 | 001000 |
| MSc/MA | 4 | 100 | 010000 |
| PhD | 5 | 101 | 100000 |
References: One hot encoding at Wikipedia
And a 2017 paper on the comparison on the effects of different encodings to neural networks in the International Journal of Computer Applications could be a good starting point: A Comparative Study of Categorical Variable Encoding Techniques for Neural Network Classifiers
-
$\begingroup$ Thank you very much but for level of character $\endgroup$ Commented Jan 27, 2018 at 9:30
-
1$\begingroup$ What do you mean by "but for level of character"? Sorry, I don't get it. $\endgroup$– oszkarCommented Jan 27, 2018 at 10:51
-
1$\begingroup$ when make the pre-processing of data(encoding) to enter the model we represent each character to zeros and ones not word $\endgroup$ Commented Jan 27, 2018 at 12:31
-
1$\begingroup$ Also, if you have $n$ unique categories (or words here), OHE results in either $n$ or $n-1$ features where as binary encoding results in only $\log_{2} n$. So if your vocabulary is $100$ words then OHE needs at least $99$ features whereas binary encoding needs only $7$ which is a major reduction in dimensionality. $\endgroup$– DanCommented Feb 8, 2018 at 8:03
-
1$\begingroup$ I just read this paper: diva-portal.org/smash/get/diva2:1259073/FULLTEXT01.pdf Unsuprisingly, one-hot encoded variables give the best precision. I think it's a good practice to only use binary encoding when we are limited by resources. $\endgroup$– neehCommented Feb 7, 2023 at 4:01