25
$\begingroup$

In one-hot encoding there is one bit reserved for each word we desire to encode.

How is multi-hot encoding different from one-hot? In what scenarios would it make sense to use it over one-hot?

$\endgroup$

3 Answers 3

33
$\begingroup$

Imagine your have five different classes e.g. ['cat', 'dog', 'fish', 'bird', 'ant']. If you would use one-hot-encoding you would represent the presence of 'dog' in a five-dimensional binary vector like [0,1,0,0,0]. If you would use multi-hot-encoding you would first label-encode your classes, thus having only a single number which represents the presence of a class (e.g. 1 for 'dog') and then convert the numerical labels to binary vectors of size $\lceil\text{log}_25\rceil = 3$.

Examples:

'cat'  = [0,0,0]  
'dog'  = [0,0,1]  
'fish' = [0,1,0]  
'bird' = [0,1,1]  
'ant'  = [1,0,0]   

This representation is basically the middle way between label-encoding, where you introduce false class relationships (0 < 1 < 2 < ... < 4, thus 'cat' < 'dog' < ... < 'ant') but only need a single value to represent class presence and one-hot-encoding, where you need a vector of size $n$ (which can be huge!) to represent all classes but have no false relationships.

Note: multi-hot-encoding introduces false additive relationships, e.g. [0,0,1] + [0,1,0] = [0,1,1] that is 'dog' + 'fish' = 'bird'. That is the price you pay for the reduced representation.

$\endgroup$
3
  • 3
    $\begingroup$ +1 Thanks Tinu - Very useful. To complete the comparisons, "label encoding" is just one number with one value per class, right? $\endgroup$
    – Josh
    May 21, 2020 at 16:41
  • 1
    $\begingroup$ Exactly! Have a look at the LabelEncoder of sklearn to get a detailed description: scikit-learn.org/stable/modules/generated/… $\endgroup$
    – Tinu
    May 21, 2020 at 16:59
  • $\begingroup$ wow great answer thanks. $\endgroup$
    – Max Power
    Mar 15, 2022 at 1:09
21
$\begingroup$

The accepted answer seems rather eccentric to me. I think that is rarely done, if ever, and will usually yield bad results.

There's a much more common, sensible use case for this. "Multi-hot encoding" doesn't seem to be a standard term, but I'm not sure there's any standard term. scikit-learn refers to a multi label binarizer.

This is simply used for multi label problems. That is, problems where more than one label can be associated with each example.

For example, say you are trying to detect whether certain types of animal are in a photo. Note that multiple types of animal can be in a single photo. Say the possible types of animal are ['cat', 'dog', 'fish', 'bird', 'ant']. A photo containing cats and dogs would be represented as [1, 1, 0, 0, 0].

$\endgroup$
-1
$\begingroup$

I too find the accepted answer likely wrong. I could not find any reference to such an encoding anywhere. In Tensorflow and in Francois Chollet's (the creator of Keras) book: "Deep learning with python", multi-hot is a binary encoding of multiple tokens in a single vector. Meaning, you can encode a text in a single vector, where all the entries are zero, except the entries corresponding to a word present in the text is one.

Please see the Tensorflow category_encoding output_mode parameter's description at https://www.tensorflow.org/api_docs/python/tf/keras/layers/CategoryEncoding#args

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.