I have two logistic regression models A and B. A is nested under B, i.e., A's features are the subset of B's features. To evaluate both models, I use 10-fold cross validation: (1) train A and B on train datasets and then test on test datasets. (2) repeat (1) 10 times. I find that B has a higher mean of test accuracy than A.
My goal is to prove significant improvement of B over A. How can I prove that the mean test accuracy of B is indeed statistically significant higher than that of A?
EDIT: @General Abrial suggested to use likelihood ratio test. However, my scenario is a bit tricky. The difference between A and B is a categorical feature. To let logistic regression handle the categorical feature, I let feature vectors of B formed by two parts: 1. features from A 2. a binary vector indicate which category the additional categorical feature takes. The length of the binary vector is very long because the number of categories is large for that feature.
The likelihood improved by B is high. But the degree of freedom in my case is also large. This causes the likelihood ratio test always concludes B is not significant.