I have been working on a logistic regression model to predict 'yeses' in a yes/no classification problem. The objective of my problem is not necessarily to predict the outcome, but it's rather to just get a better understanding of my variables and how they influence the outcome.
For example, I want to say that feature X is 2.2 more likely to achieve 'yes' then my reference level, and Y feature Y is 2.5 less likely to achieve 'yes', etc.
I did my model, received the output, and I know how to read the coefficients, but I want to know if there are any tests I can do to confirm the confidence of the coefficients.
I did a few things already: 1. checked the P-value of the summary (focused on the ones <.05) 2. Ran each model predictor by predictor to look at the the change in prediction, BIC, AIC, AUC, confusion matrix, ROC curve, etc. 3. I tried to do a chisq test of independence on my independent variables (since they are all categorical), but I'm getting weird results with R saying the results may not be accurate and the result is p_value <.05. The contingency table do show some very low frequency so I'm wondering if that has to do with anything. 4. Did a partial dependency plot....which seems to show opposite direction of my coefficient, but that may be just a software problem on my part. 5. the original data is unbalanced, so I 'up-sampled' it in R. 6. Accuracy is about 70%, AUC is about 79%
Besides these things, are there anything else that I can do? Basically, if I say that a feature is 2.5x more likely to vote 'yes', I want to make sure that is more or less correct.
Right now, I feel like i"m just taking the output at face value and it's a bit uncomfortable.