Presenting Logistic Regression Results (Imbalanced Data, Small Sample Size) I have an imbalanced data set of 300 observations with an adverse event rate of 8%. I have 4 features that I believe to be relevant based on expertise in the field. I am interested only in inference (not prediction) and assessing potential relevance of one of the features for protection against the adverse event, so I'm using logistic regression which has good interpretability. This is not meant to be a decisive paper since I'm aware of the limitations of the data, merely to elucidate potential for further research into the field.
1) How should these results - positive or negative - be presented in a paper? Do I need more to present more than odds ratios with associated confidence intervals and p-values?
2) Since the event rate is low and sample size relatively small (and thus underpowered logistic regression), is cross-validation applicable? It seemed like it might give misleading results. If I were doing machine learning/prediction, I'd re-sample to balance (oversample or undersample with something like SMOTE), but I'm also not interested at all in accuracy or other metrics like F1, just the parameter estimates.
In general, what are my options for assessing the results of logistic regression given that I am not interested in prediction?
 A: As you are not interested in using your model estimates to make predictions, I don't thin cross-validation is relevant in your case (You don't really seem to care about data over-fitting).
I think you've already pointed out the major limitations of the data - Underpower means that some your results are likely to suffer from type II error (i.e., not rejecting the hypothesis of coeff nullity while you should).
Model log-likelihood in itself is not very informative, except if you use it to compare diff model specifications. For example, you could compare the perf of your "target" model with an empty model (i.e., intercept only), it would tell you something about explanatory power of your independent variables.
You could also look at alternative to R-squared in context of logistic regression (e.g., McFadden Pseudo R2 => McFadden's Pseudo-R2 Interpretation).
You could also compute the % of correctly predicted events (0/1) using your model estimates and compare it to pure chance level (~50%) - It would also tell you something "interesting" about your model.
A: This looks like a situation where going Bayesian makes sense. You have very few positive cases and if you're only interested in the parameter estimates to suggest some directions for future work, a full presentation of the uncertainty associated with those estimates is going to be important, IMO. You might also consider a little bit of regularisation with e.g. a Cauchy prior on the non-intercept parameters.
rstanarm will do Bayesian LR for you in one line of code and it will probably take less than a second to run for this example. If you want some help with it, include your data somewhere and I'll have a look.
