I'm wondering whether it's necessary to use one-hot if a feature has only two values but not {0,1}.
I'm also wondering whether there is a good way to reduce the number of features after one-hot. Is PCA a good method？
$\begingroup$
$\endgroup$
3
-
2$\begingroup$ In statistics, "one-hot coding" is known as dummy variables or indicator variables coding. It is used to recode a nominal categorical variable into a set of binary variables belonging together. $\endgroup$– ttnphnsCommented Feb 20, 2017 at 6:57
-
$\begingroup$ Standard PCA is not for categorical data therefore running it on dummy variables is mostly senseless. Categorical PCA or Correspondence analysis (see Tim's answer) is an option. However, if your data are all dichotomous variables which you can understand as ordinal: "attribute present" vs "attribute absent" then you may run PCA on it: PCA is allowed to do on binary variables; recode 1->0, 2->1 and you will be happy. Note that this is not dummy recoding (in a dummy recoding, the categorical variable is seen as nominal and you create two binary variables out of one dichotomous variable). $\endgroup$– ttnphnsCommented Feb 20, 2017 at 7:06
-
$\begingroup$ Might want to read also stats.stackexchange.com/q/215404/3277, stats.stackexchange.com/q/16331/3277. $\endgroup$– ttnphnsCommented Feb 20, 2017 at 7:07
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
One hot encoding is a method to deal with the categorical variables. Now coming to your problem your data has only { 1,2 } you can use it as it is but using {1,2} imparts ordinal characteristics to your data like 1<2 and if your model is sensitive like random forest or something like that then it will surely effect your output. So you can use one-hot if you want.
There are many methods to reduce the sparsity like PCA (Principal Component Analysis) or LDA (Linear Discriminant Analysis) etc you can choose according to your need.
Yes PCA is a good method for dimension reduction.
$\begingroup$
$\endgroup$
2
The answer depends on the type of algorithm you are using. For a traditional predictive algorithm, such as regression, CART, LDA, etc., it will either have no effect, or, cause an error (due to perfect multicollinearity).
If you are using a technique like cluster analysis, you may get a different answer, as the one hot coding will have the effect of changing the weighting of the variables.
As regards using PCA for feature reduction after one-hot programming, this is quite common in non-academic work, although some PCA algorithms will give you an error message if you don't exclude one of the variables for each categorical variable. The more orthodox solution would be to use a dimension reduction technique that can accommodate categorical data or mixed data (e.g., Multiple Correspondence Analysis, non-linear PCA, or HOMALS)
-
$\begingroup$ +1. One notion, however. Homogeneity analysis HOMALS, as I'm aware, is a synonym to MCA. Non-linear categorical PCA has acronyms CatPCA or PRINCALS. When optimal scaling in it is set "multiple nominal" for all the input variables it turns into MCA actually. $\endgroup$– ttnphnsCommented Feb 20, 2017 at 7:18
-
$\begingroup$ I have read people saying this before. However, most implementations in my experience are different. Typically, MCA only includes categorical variables, nonlinear PCA often permits multiple data types (e.g., in SPSS), whereas HOMALs, if my memory serves me correct, also permits ordinal variables. $\endgroup$– TimCommented Feb 20, 2017 at 7:35
$\begingroup$
$\endgroup$
If these features are not ordinal i.e. 2 is not greater or lesser than 1 then it makes sense to do one hot encoding . As long as the values are nominal it does not matter if they are {0,1} or {1,2} or {3,4} or {male,female}.
PCA is good for dimentionality reduction but if your end aim is classification I would rather use methods like LDA's which also take your labels into account.
Also, in my research ive observed that if you have enough data and your aim is to do classification then you should go for autoencoders (i was working with audio but I suspect it is equally well for other data sets also as per the original authors, it learns PCA like features and is a better basis as seen by many others). However what constitutes as "Enough data" is dependent on the problem .
They have an added benefit as most times a non linear transformation of data may work for you, but with PCA you are limiting yourself to linear transformations and that too without taking advantage of the labels also if the data is too big it becomes difficult to really do the matrix operations involved ,on the other hand auto encoders can be batch processed. Autoencoders can be fine tuned and thus overall you end up achieving a better basis for classification.
That being said ,its good to experiment with PCA....It may work better or not, all depends on your problem.You might want to check out this thread.
