What is the name of the operator that takes a categorical vector and transforms it to the binary representation using one-hot encoding? I am wondering since I am writing a scientific paper and need a proper name for that.
$\begingroup$
$\endgroup$
2
-
4$\begingroup$ Dummy encoding is another name. In machine learning, everyone just uses the one simple type so it's pretty clear what this is, but there are other types of contrast coding (another name) with minus ones and other ideas, that perform a similar role, used in statistics, and so you can be a bit more specific. $\endgroup$– GijsCommented Oct 19, 2017 at 19:57
-
7$\begingroup$ In statistics and data analysis, long before machine learning, this type of categorical encoding has been known as dummy variables aka indicator type contrast variables. $\endgroup$– ttnphnsCommented Oct 19, 2017 at 20:37
Add a comment
|
6 Answers
$\begingroup$
$\endgroup$
6
Statisticians call one-hot encoding as dummy coding. As others suggested (including Scortchi in the comments), this is not exact synonym, but this is the term that would be usually used for the 0-1 encoded categorical variables.
See also: "Dummy variable" versus "indicator variable" for nominal/categorical data
-
3$\begingroup$ Duh!! Can't believe I forgot that. I also refer to them as indicators. $\endgroup$ Commented Oct 19, 2017 at 21:40
-
2$\begingroup$ I don't think "dummy coding" is a good synonym. It's used either in a general sense to mean representing a categorical variable with a set of numeric variables, or for "reference-level coding" as distinct from "one-hot encoding", e.g. in Problems with one-hot encoding vs. dummy encoding. "Level-means coding" (see Is there something called “mean coding” (like dummy coding & effect coding) in regression models?) denotes one-hot encoding, but connotes a linear model context perhaps too ... $\endgroup$ Commented Oct 20, 2017 at 10:29
-
2$\begingroup$ ... strongly for general use. $\endgroup$ Commented Oct 20, 2017 at 10:29
-
3$\begingroup$ I've never seen a definition per se, but to me dummy variables in statistics always implies the coding of N factors with (N-1) variables whereas one-hot encoding will code N factors with N variables. This difference is tremendously important in practice. If one use one-hot encoding for regressions, one would get nonsense because of the dependence of the variables ! $\endgroup$– mehCommented Oct 20, 2017 at 13:17
-
2$\begingroup$ @aginensky Although people should certainly pay attention to exactly what variables they have, a good regression routine won't produce nonsense in that circumstance: it will just omit one predictor and tell you so. $\endgroup$– Nick CoxCommented Oct 20, 2017 at 13:22
$\begingroup$
$\endgroup$
It depends on your target audience.
As Tim said, statisticians call it dummy coding, and that's what I would expect to see when describing something like a regression model. "Dummy coded variables were included to adjust for the store's location." I think calling it a one-hot encoding would seem slightly strange here.
However, as another Tim also said, one-hot encoding is fairly common in the machine learning literature. It faintly implies the existence of nodes (as in a neural network), physical wires (in a device), or something like that, at least to me.
Formally, I guess you are applying a set of indicator functions $\mathbb{I}_X$, but that's probably way too formal outside of a proof.
answered Oct 19, 2017 at 21:24
$\begingroup$
$\endgroup$
The term comes from electronics engineering. Just think who would call 1 "hot"? Only those who work with electricity, where "hot" or "live" means there's electrical potential on the wire. "One hot" refers to the circuit design where discrete electrical signal level on one wire would be decoded into hot/cold on a set of wires. I suppose some machine learning folks with EE background found the analogy compelling.
In econometrics and statistics you may encounter dummy or indicator variables, which are quite similar because these are used to represent distinct categories with their distinct indicators. There's a subtle difference though. For instance, you make K-1 dummies for K categories, because the base category corresponds to all dummies set to 0. In contrast, I think that in one hot encoding you have K wires, where the base category will have its own wire (variable).
$\begingroup$
$\endgroup$
2
I'm statistically trained, and have recently heard of "one-hot encoding" in machine learning/comp sci lit. I've usually just referred to the one-hotted matrix as a design matrix/data matrix/design frame.
-
$\begingroup$ Do you have a reference that I could cite for that? I am writing a scientific publication and would like to make clear about this method for all readers since the paper is not for ML community but broader. $\endgroup$– fractileCommented Oct 19, 2017 at 20:02
-
$\begingroup$ Can't say I've ever heard "one-hotted" as a verb. But I similarly come to this from a mathematical/statistical direction. (Google results on "one-hotted" are interesting - I get a mixture of the machine learning meaning and people talking about "one hotted-up car".) $\endgroup$ Commented Oct 20, 2017 at 13:06
$\begingroup$
$\endgroup$
8
In physical sciences and engineering, it's called the (generalized) Kronecker delta.
In simplest form, the Kroneker delta's defined as $$ \begin{align*} {\delta}_{i,j} {\equiv} \begin{cases} 1 &\text{if} & i=j \\ 0 &\text{else} \end{cases} \end{align*}, $$though this trivially generalized to$$ \begin{align*} {\delta}_{\left[\text{condition}\right]} {\equiv} \begin{cases} 1 &\text{if} & \left[\text{condition}\right] \\ 0 &\text{else} \end{cases} \end{align*}. $$
So, "${\delta}_{i{\in}\text{category}}$" will tend to be read as$$ \begin{align*} {\delta}_{i{\in}\text{category}} {\equiv} \begin{cases} 1 &\text{if} & i{\in}\text{category} \\ 0 &\text{else} \end{cases} \end{align*}, $$ which most authors would tend to truncate to "${\delta}_{i}$", if the category is obvious from context.
The Kronecker delta is really useful in Sigma/Pi/Einstein/etc. notations since it allows for terms to be specified conditionally.
Just to relate this to common programming structures, the Kronecker delta's condition?1:0, where ?: is the conditional operator.
As a tangential note, I'd encourage authors to abandon the old-fashion ${\delta}_{i,j}$ in favor of the generalized equivalent, ${\delta}_{i=j}$. There's no advantage to the old-fashion notation, while the generalized notation's more explicit and extensible.
-
$\begingroup$ I don't see the link here. One hot decodes one variables into a set for each state of the variable. How is Kronecker delta used in this application? $\endgroup$– AksakalCommented Oct 20, 2017 at 13:19
-
$\begingroup$ @Aksakal This post gives an example of encoding a category "CompanyName" with possible values "VW", "Acura", and "Honda", which becomes three $\left\{0,1\right\}$ variables by those value names, where @Tim's answer calls those "dummy variables". These are the same thing as the Kronecker deltas ${\delta}_{\text{VW}}$, ${\delta}_{\text{Acura}}$, and ${\delta}_{\text{Honda}}$. $\endgroup$– NatCommented Oct 20, 2017 at 14:58
-
$\begingroup$ @Aksakal I prefer the generalized notation, but in the old notation, ${\delta}_{i,j}$, it'd be ${\delta}_{\text{CompanyName},\text{VW}}$, etc.. $\endgroup$– NatCommented Oct 20, 2017 at 15:01
-
$\begingroup$ The dummy works like this: you have the variables called $VW$ and $ACURA$. Your observations are $i=1..N$, so you get the values $VW_i$ and $ACURA_i$, both are zero when the car is HONDA. Note, that here $i$ is not the make of the car, it's the number of the observation. I still don't see how do you connect this to Kronecker delta. Say, if the original variable was $CAR_i$, then the delta would work like $VW_i=\delta(CAR_i,VW)$ $\endgroup$– AksakalCommented Oct 20, 2017 at 15:02
-
$\begingroup$ @Aksakal The value that you're calling "${VW}_{i}$" is ${{\delta}_{\text{VW}}}_i$ or ${\delta}_{i{\in}\text{VW}}$. If $i$ is a VW, then it'd be $1$; otherwise, it's $0$. $\endgroup$– NatCommented Oct 20, 2017 at 15:07
$\begingroup$
$\endgroup$
Pattern Recognition and Machine Learning by Christopher Bishop uses the term $1$-of-$K$ scheme.
Here is a quote from the book,
Binary variables can be used to describe quantities that can take one of two possible values. Often, however, we encounter discrete variables that can take on one of $K$ possible mutually exclusive states. Although there are various alternative ways to express such variables, we shall see shortly that a particularly convenient representation is the $1$-of-$K$ scheme in which the variable is represented by a $K$-dimensional vector $\textbf{x}$ in which one of the elements $x_k$ equals $1$, and all remaining elements equal $0$. So, for instance if we have a variable that can take $K = 6$ states and a particular observation of the variable happens to correspond to the state where $x_3 = 1$, then $\textbf{x}$ will be represented by,
$\textbf{x} = (0, 0, 1, 0, 0, 0)^{T}$
