3
$\begingroup$

I have 2 categorical variables with 8 unordered categories and multiple numerical variables and I want to train a logistic regression model. I want to test the independence between all my predictor variables, and remove the ones that are dependent, meaning that they are redundant in my model.

Is there any universal statistical test to test the independence between quantitative and categorical predictors? For two quantitative variables I know I could use correlation tests, and for two categorical, the $\chi^2$ test, but what about a quantitative and a categorical variable?

$\endgroup$
2
  • $\begingroup$ There are a number of possibilities. Do we know anything further about them? How many categories? Are they ordered? Is one or the other variable regarded as a response? Is there independence within variables? Are there restrictions on the distribution of the quantitative variable? Are there particular kinds of dependence we're interested in (n.b. correlation is checking a very particular kind of dependence)? $\endgroup$
    – Glen_b
    Jun 1, 2013 at 17:10
  • $\begingroup$ I have 8 categories and not ordered. In fact I have 2 categorical variables and multiple numerical variables and I want to train a logistic regression model. I want to test the independence between all my variables (only predictors), and remove the ones that are dependent, meaning that they are redundant in my model. $\endgroup$ Jun 1, 2013 at 20:03

4 Answers 4

3
$\begingroup$

I have 8 categories and not ordered. In fact I have 2 categorical variables and multiple numerical variables and I want to train a logistic regression model. I want to test the independence between all my variables (only predictors), and remove the ones that are dependent, meaning that they are redundant in my model.

It would have helped you if you'd started with this information.

1) Pairs of variables with highly significant correlations (i.e. very small p-values) may both be needed in a model - indeed they may be very far from "redundant"; a significant pairwise correlation tells you very little about that. Indeed with large samples even trivially small correlations may be highly significant. Hypothesis tests answer the wrong question here (they don't tell you about the impact of the correlation on your inference).

2) If measures of association with categorical variables don't measure the same kind of dependency that matters in your model, it's not telling you what you need to know about.

3) It's quite possible for variables to be pairwise not all that correlated but highly dependent in larger groups; you can check every pairwise correlation and find it's almost zero, yet still have redundant variables across the whole set.

Your entire approach is simply misguided - it might help, but it might utterly fail to avoid redundancy and it might get you to throw out important variables for no good reason at all. You approach tells you less than you might think about redundancy of your variables.

You need to consider your variables as a collection. The right sort of thing to do is check the condition of your $X$-matrix, or something related to it. Sometimes people use things like variance inflation factors, or how completely each $X_j$ is predicted by the collection of previous (or even all other) $X$'s or various other measures. This sort of checking is fairly standard while doing regressions and GLMs.

$\endgroup$
2
$\begingroup$

This is mostly just to support and explicate the points @Glen_b has made.

The fact that variables are highly correlated does not necessarily mean they are redundant and that you want to throw one of them out. Consider the data below:

enter image description here

For these data, $x_1$ and $x_2$ are correlated $r = .99337$. If you were to fit a logistic regression model regressing $y$ on $x_1$ only, the area under the curve would be exactly $.5$. If you regressed $y$ on $x_2$ only, $AUC = .5644$. But if you regressed $y$ on both, then $AUC = 1.0$.

$\endgroup$
1
$\begingroup$

Yes, ANOVA. There may be more than one quantitative (in this case it is called MANOVA) and also more than one qualitative variables (in this case it is called multi-way ANOVA).

$\endgroup$
2
  • $\begingroup$ I know ANOVA to test the differences in the means between groups, but never seen to test independence, it is not the same no? Plus some of the assumptions for anova are not met (normality for example). Could you point me to an example? $\endgroup$ Jun 1, 2013 at 20:44
  • $\begingroup$ You are right, but note that testing whether any quantitative response is independent on any qualitative factor is the same as testing whether responses in groups are different (i.e. factor variable has any influence on response). If some assumptions of ANOVA are not met, you can either transform data or use nonparametric version of ANOVA (in case of one qualitative and one quantitative variable, it is Kruskal-Wallis test). $\endgroup$
    – sitems
    Jun 1, 2013 at 20:51
1
$\begingroup$

This test is necessarily highly elusive for reasons well explained by others.

But see the paper Link for a recent review of general measures of dependence.

But your specific problem can be answered trivially, if possibly facetiously. To work out whether predictors will be helpful in logistic regression, try fitting a model that includes them. Why be indirect about it?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.