1
$\begingroup$

I would like to know what might be the side effects of combining several features into one single feature for classification tasks. Imagine I have two variables with the following domains:

  • A is binary (in {0,1})
  • B are integers
  • C in [0, 1)

We could feed the classifier with tuples (a, b, c), or we can represent the three features using one real number: the variable A can be expressed with the sign of the number, the integer part is B and the decimal part is C:

X = (B + C) * (1 if A == 0 else -1)

It seems this seems like a good idea to reduce the number of dimensions of the space, but what are the dis/advantages of using X instead of A, B and C? In which cases is this recommendable and in which cases this is not a good idea?

$\endgroup$
2
$\begingroup$

Yes, you can expect side effects. But it will depend on the algorithm you feed the data to. Somehow, your compression is loss-free and there exist a (non linear) mapping between (A,B,C) and X. As the mapping is non linear, some methods might recover it, others may not.

A decision tree might (theoretically) produce the exact same output with both encoding. Indeed if(X>0) is equivalent to if(A==1), if(X)>3.5 is equivalent to if(B>3 and C>0.5)...

On the other hand, you enforce an order on the values of your predictor that may not be relevant (and that was not here in the first place). It could harm performance in the case of a linear regression.

Besides, the metric on your input space is totally different now. Say $x_1=(1,3,0.3),x_2=(0,3,0.3)$. The distance in the original space is 1 and 6.6 in the mapped space. Likewise, you can show that some pairs will look closer in a space and not in the other. This will certainly affect kernelized methods, KNN...

It seems this seems like a good idea to reduce the number of dimensions of the space

It depends on how you do it. Linear dimension reduction methods (like PCA or Random Projection) usually provide good results. There are plenty of non linear methods as well. But not every dimension reduction is fruitful.

You could map $\mathbb{R}$ and $\mathbb{R^2}$ (even $\mathbb{R^n}$) considering the alternating sequence of decimals of each number. This would turn any $n$ dimensional data set into a one dimensional dataset. However, it is highly unlikely that you will get better performance with such a representation of the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.