I'm new in data science so please sorry if this is silly question to be asked. While building a predictive model on machine learning using logistic regression I got stuck in statistics tests needed. I have some doubts in statistics tests and following are the questions:

1) In order to check independency in between predictors certain statistics tests are requires as:

i) continuous vs continuous(correlation test)
ii) categorical vs continuous(ANOVA test)
iii) categorical vs categorical(chi-square test)
But want to ask,
a) what would I concluded from these tests results?
b) suppose if two predictors are dependent then what to do next?
c) should I need to keep the dependent predictors,if yes then why?
d) In one the test chi square value is 18.35 and p-value is 0.08. now what i can concluded from these value I know that P-value is greater the 0.05 but don't get what it represents?


If I understood your question correctly, you want to test for dependencies among predicting variables. Such tests are commonly known as looking for multicollinearity. A simple way to look for multicollinearity is calculating variance inflation factors; basically regressing each predicting variable on all the other predicting variables. In R, in case you use this as software package, that may be done with the package "car" and works well for logistic regression. If Variance inflation is small, say smaller than 3, there is nothing you need to worry about. If it is much larger, in particular above 10, it may be a problem.

Large correlation among confounding variables (i.e. predictors you control for but for which you are not interested in their coefficients) can suffer from multicolinearity without causing troubles for your analysis. However, if your predictive variable of interest suffers from multicollinearity, its predicted coefficient may be inflated (i.e. too large, too small or even with the wrong sign) - and that could be a problem for your findings. As an advanced tool in those situations, ridge regression may help. Finally, if you are only interested in the prediction (i.e. the independent variable; the classic situation in "machine learning" compared to "traditional statistics"), multicollinearity is not such a big issue - it "only" increases the confidence interval of your predicted independent variables.

Please note that testing pairs of predictors using classic correlation tests is not the preferable approach to detect multicollinearity. That is because it is conceivable that for instance three predicting variables may jointly be much stronger correlated with a fourth variable than each of them is individually with this fourth variable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.