How to understand if 2 very correlated predictors influence the output?

Question

I have a binary output $y$ and a small set of predictors $x_1, x_2,\dotsc$

Two of these predictors are very correlated. It is known that one of them has influence on the output we want to know if also the other is relevant or not.

Is there any typical way to solve this problem?

EDIT

I have tried various methods to solve this problem but I would like to know if there is a better one.

1. run a simple logistic regression and look at the p-value

2. ortogonalize the input variables define $x_2^{new} = x_2 - x_1\cdot x_2x_ 1$ (like the gram schmidt algorithm)

3. Compute the AUC error using bootstrap and check if the model, where both the variables are present, lead to a lower AUC error. (The models are all incredibly weak and all the prediction performances are very bad)

I need to run this kind of analysis with different combinations of input and various output values. I will so not discuss the result obtained with these methods. I would like to know if there is a standard way to do that.

MY BACKGROUND: I am a mathematician but I have a good knowledge of machine learning. I know a little bit of statistics as well but not too much.

Answer 1

So you have set the rest of the model and you only want to test $x_2$ effect each time? Also, you are interested in an inferential statistical approach (which is, using hypothesis testing)?

If so, the simplest thing to do is to fit both model with and without $x_2$ and compare them trough a likelihood ratio test. Also looking at the p-value of $x_2$ (which comes from a Wald test instead) after ortogonazing it, should work fine, but likelihood ratio is a bit more reliable, and it doesn't even need ortogonalization.

Answer 2

Typical approach is to use PCA and transform original features in uncorrelated components and select components that explain most of the variance. Also one of highly correlated features may be removed in preprocessing. This is called filter feature selection.