How is One-Hot Encoding interpreted by an Algorithm?

Question

I'm new to machine learning, and just learned about the use of one-hot encoding as a method of passing a categorical variable as an input into a machine learning algorithm. As I understand it, one of the advantages of one-hot encoding is that it allows the algorithm to learn separate weights for each possible value of the categorical variable.

My questions are the following: If the model is made to learn a separate weight for each input feature, how can it learn N weights when it comes to a one-hot encoded feature of N possible values, and then only apply the relevant one? Isn't this the same as turning the categorical variable into N individual boolean features?

Thanks in advance!

Answer 1

Yes, you are absolutely right. One-hot-encoding is the same as "turning the categorical variable into N individual boolean features" and is called "creating dummy variables" in statistics. The main purpose of doing this is that you can easily manipulate a model/neural network/whatever else you are using by using matrix algebra.

E.g. if you have a class that is either "cat", "dog" or "bird", then it's hard to see how to apply linear algebra. So, instead we represent this as $x=(1,0,0)^T$, $(0,1,0)^T$ or $(0,0,1)^T$, where $^T$ denotes transposing. Then it becomes easy to have a second vector of weights or coefficients $\beta = (\beta_1, \beta_2, \beta_3)^T$, and to write that the mean outcome is $\mu = x^T\beta$. That is a more convenient notation that saying that the mean outcome for cats is $\beta_1$, for dogs $\beta_2$ and $\beta_3$ for birds, but more importantly computers can easily handle this using linear algebra in the same way as for continuous predictors.