I have two logistic regression models A and B. A is nested under B, i.e., A's features are the subset of B's features. To evaluate both models, I use 10-fold cross validation: (1) train A and B on train datasets and then test on test datasets. (2) repeat (1) 10 times. I find that B has a higher mean of test accuracy than A.

My goal is to prove significant improvement of B over A. How can I prove that the mean test accuracy of B is indeed statistically significant higher than that of A?

EDIT: @General Abrial suggested to use likelihood ratio test. However, my scenario is a bit tricky. The difference between A and B is a categorical feature. To let logistic regression handle the categorical feature, I let feature vectors of B formed by two parts: 1. features from A 2. a binary vector indicate which category the additional categorical feature takes. The length of the binary vector is very long because the number of categories is large for that feature.

The likelihood improved by B is high. But the degree of freedom in my case is also large. This causes the likelihood ratio test always concludes B is not significant.


I suggest you to do it differently: you can transform your binary vector into one single quantitative variable using for example the technique of Weight of Evidence. This technique is described a.o. in the following paper (section 3.2.3): http://www.sciencedirect.com/science/article/pii/S0167923615000275. After having made this transformation, you don't have this problem of many categories and high df.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.