Relationship between related continuous and categorical variables

Question

I'm trying to run a regression model for data with 150 features. I'm eliminating redundant features to prevent over-fitting and produce a model that is easier for illustration. There are two features that I'm not sure how to handle, a real variable and a categorical variable.

x1 is a measure of distance

c1 depending on a threshold (say x1 > 100), c1 is yes or no (encoded 1 and 0 respectively)

These two variables are obviously highly related (I'm actually not sure why both were present on the dataset). I would like to get rid of one if possible.

1) Is it safe to get rid of one of them? Could the fact that one is categorical have some unforeseen effect? And which one should I get rid of?

2) Is there a mathematical way to show that these two variables are highly related? (I've tried correlation coefficient, which didn't work)

Answer 1

You should not include both variables in the model. It will almost always be better to include a continuous variable than a dichotomy formed from that continuous variable, so, unless there is something unusual going on, you should keep the continuous one. You might have nonlinear relationships between it and the dependent variable. If so, you can use polynomials or splines.

It shouldn't really be necessary to show that the two are correlated; what do you mean the correlation coefficient "didn't work"? Here is how you could do it in R, with some made up data:

set.seed(1234)
x1 <- rnorm(1000, 100, 10)  #Random normal, mean = 100, sd = 10
c1 <- x1 > 100  #Dichotomy as in your data
cor(x1, c1)  #0.79

Answer 2

To get a sense of correlation and covariation you can see the following approaches :

a) Graphically - Study the variation of both the variables by plotting a graph of both of them against the dependent variable. This gives you a visual idea but is not preferred when you have a large number of variables since its not a scalable approach.

b) Statistically - Compute the variance inflation factor (VIF), it is generally available as an output to several packages. If two or more variables are highly correlated both of them will have a high VIF (more than 2.5), you can remove variables by either using your intuition of checking for their significance (p-value). Once you remove one (or more) of the correlated variables, you will see these metrics taking acceptable values.

Answer to 1 - Remove it only if you can graphically or statistically show a good degree of co-variation, it is not a good idea to remove it just on the basis of your hunch.

Answer to 2 - Use the methods given above.

Let me know if you need further clarification.