2
$\begingroup$

I'm new to machine learning, and just learned about the use of one-hot encoding as a method of passing a categorical variable as an input into a machine learning algorithm. As I understand it, one of the advantages of one-hot encoding is that it allows the algorithm to learn separate weights for each possible value of the categorical variable.

My questions are the following: If the model is made to learn a separate weight for each input feature, how can it learn N weights when it comes to a one-hot encoded feature of N possible values, and then only apply the relevant one? Isn't this the same as turning the categorical variable into N individual boolean features?

Thanks in advance!

$\endgroup$
3
$\begingroup$

Yes, you are absolutely right. One-hot-encoding is the same as "turning the categorical variable into N individual boolean features" and is called "creating dummy variables" in statistics. The main purpose of doing this is that you can easily manipulate a model/neural network/whatever else you are using by using matrix algebra.

E.g. if you have a class that is either "cat", "dog" or "bird", then it's hard to see how to apply linear algebra. So, instead we represent this as $x=(1,0,0)^T$, $(0,1,0)^T$ or $(0,0,1)^T$, where $^T$ denotes transposing. Then it becomes easy to have a second vector of weights or coefficients $\beta = (\beta_1, \beta_2, \beta_3)^T$, and to write that the mean outcome is $\mu = x^T\beta$. That is a more convenient notation that saying that the mean outcome for cats is $\beta_1$, for dogs $\beta_2$ and $\beta_3$ for birds, but more importantly computers can easily handle this using linear algebra in the same way as for continuous predictors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.