1
$\begingroup$

I have a feature vector X in a regression problem, where one of the features X1 is categorical (genre) with 47 categories. There is also another feature X2 which is continuous (# subscribers) but takes fixed value on each of the category. If I dummy encode X1 into vectors, X2 can be perfectly expressed as a linear combination of these vectors. I have few questions:

  1. Can X1 be completely dropped, since the uniqueness of X2 ensures that information is not lost?
  2. If I drop one of the dummy variables, is there a loss of mapping between X1 and X2 (and so I keep both X1 and X2) ?
  3. Is it a good idea if I multiply X1 vectors by X2, so that instead of 1s they take X2 values, so information from both variables is retained (and hence not dropping the first dummy variable)?
$\endgroup$

1 Answer 1

0
$\begingroup$

If one of your features X2 can be expressed by another one X1, you can drop X2 without loss of information. More precisely: if you can predict for each new measurement (X1, X2, X3, ..., Xn) the exact value of X2 from X1, X2 doesn't carry any new information and can be dropped; at least from the information-theoretical point of view.

Think about it the other way around: if there was a difference, then it would make sense to create a lot of new variables that are just deterministic functions of those that are already present, and each one would improve the amount of contained information.

Having said that, there is something like "feature engineering": sometimes (or should I rather say "always") it makes sense to replace a feature / several features with a transformation / several transformations of it/them. E.g., think of scaling or standardization. That could make your regression algorithm work better, but the details depend on the type of data and regression you use and are usually part of the documentation of that algorithm.

$\endgroup$
1
  • $\begingroup$ If you are satisfied with the answer, please accept it. If not, you could consider leaving a comment detailing what you are missing. $\endgroup$
    – frank
    Mar 19, 2022 at 6:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.