3
$\begingroup$

I have 2 categorical variables with 8 unordered categories and multiple numerical variables and I want to train a logistic regression model. I want to test the independence between all my predictor variables, and remove the ones that are dependent, meaning that they are redundant in my model.

Is there any universal statistical test to test the independence between quantitative and categorical predictors? For two quantitative variables I know I could use correlation tests, and for two categorical, the $\chi^2$ test, but what about a quantitative and a categorical variable?

$\endgroup$
  • $\begingroup$ There are a number of possibilities. Do we know anything further about them? How many categories? Are they ordered? Is one or the other variable regarded as a response? Is there independence within variables? Are there restrictions on the distribution of the quantitative variable? Are there particular kinds of dependence we're interested in (n.b. correlation is checking a very particular kind of dependence)? $\endgroup$ – Glen_b -Reinstate Monica Jun 1 '13 at 17:10
  • $\begingroup$ I have 8 categories and not ordered. In fact I have 2 categorical variables and multiple numerical variables and I want to train a logistic regression model. I want to test the independence between all my variables (only predictors), and remove the ones that are dependent, meaning that they are redundant in my model. $\endgroup$ – juniordataminer Jun 1 '13 at 20:03
3
$\begingroup$

I have 8 categories and not ordered. In fact I have 2 categorical variables and multiple numerical variables and I want to train a logistic regression model. I want to test the independence between all my variables (only predictors), and remove the ones that are dependent, meaning that they are redundant in my model.

It would have helped you if you'd started with this information.

1) Pairs of variables with highly significant correlations (i.e. very small p-values) may both be needed in a model - indeed they may be very far from "redundant"; a significant pairwise correlation tells you very little about that. Indeed with large samples even trivially small correlations may be highly significant. Hypothesis tests answer the wrong question here (they don't tell you about the impact of the correlation on your inference).

2) If measures of association with categorical variables don't measure the same kind of dependency that matters in your model, it's not telling you what you need to know about.

3) It's quite possible for variables to be pairwise not all that correlated but highly dependent in larger groups; you can check every pairwise correlation and find it's almost zero, yet still have redundant variables across the whole set.

Your entire approach is simply misguided - it might help, but it might utterly fail to avoid redundancy and it might get you to throw out important variables for no good reason at all. You approach tells you less than you might think about redundancy of your variables.

You need to consider your variables as a collection. The right sort of thing to do is check the condition of your $X$-matrix, or something related to it. Sometimes people use things like variance inflation factors, or how completely each $X_j$ is predicted by the collection of previous (or even all other) $X$'s or various other measures. This sort of checking is fairly standard while doing regressions and GLMs.

$\endgroup$
2
$\begingroup$

This is mostly just to support and explicate the points @Glen_b has made.

The fact that variables are highly correlated does not necessarily mean they are redundant and that you want to throw one of them out. Consider the data below:

enter image description here

For these data, $x_1$ and $x_2$ are correlated $r = .99337$. If you were to fit a logistic regression model regressing $y$ on $x_1$ only, the area under the curve would be exactly $.5$. If you regressed $y$ on $x_2$ only, $AUC = .5644$. But if you regressed $y$ on both, then $AUC = 1.0$.

$\endgroup$
1
$\begingroup$

Yes, ANOVA. There may be more than one quantitative (in this case it is called MANOVA) and also more than one qualitative variables (in this case it is called multi-way ANOVA).

$\endgroup$
  • $\begingroup$ I know ANOVA to test the differences in the means between groups, but never seen to test independence, it is not the same no? Plus some of the assumptions for anova are not met (normality for example). Could you point me to an example? $\endgroup$ – juniordataminer Jun 1 '13 at 20:44
  • $\begingroup$ You are right, but note that testing whether any quantitative response is independent on any qualitative factor is the same as testing whether responses in groups are different (i.e. factor variable has any influence on response). If some assumptions of ANOVA are not met, you can either transform data or use nonparametric version of ANOVA (in case of one qualitative and one quantitative variable, it is Kruskal-Wallis test). $\endgroup$ – Miroslav Sabo Jun 1 '13 at 20:51
1
$\begingroup$

This test is necessarily highly elusive for reasons well explained by others.

But see the paper http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ss/1359468411 for a recent review of general measures of dependence.

But your specific problem can be answered trivially, if possibly facetiously. To work out whether predictors will be helpful in logistic regression, try fitting a model that includes them. Why be indirect about it?

$\endgroup$
0
$\begingroup$

No, there is no unified test that would be adequate in every scenario. The appropriate test is driven by the question being asked. For instance, "What is the reduction in colon cancer risk associated with drinking one cup of green tea daily in a case control sample of Japanese women over the age of 50?" or "How much more likely are white males to vote republican than other voters in the top ten most populated census tracts?" or "Is there a relationship between difference in proportions of individuals living below poverty line and number of years of education in a matched cohort of twins?" Each of these requires an understanding of how the data are coded, what the adequate association measure is, and the sampling design. Case control, case cohort, stratified samples, and ecological studies are examples of study designs. Relative risk, risk difference, odds ratios, population attributable risk are examples of association measures simply for binary outcomes. There is too many interesting questions for a statistician to have a tool box with just one tool that measures them all.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.