3
$\begingroup$

I have an imbalanced data set of 300 observations with an adverse event rate of 8%. I have 4 features that I believe to be relevant based on expertise in the field. I am interested only in inference (not prediction) and assessing potential relevance of one of the features for protection against the adverse event, so I'm using logistic regression which has good interpretability. This is not meant to be a decisive paper since I'm aware of the limitations of the data, merely to elucidate potential for further research into the field.

1) How should these results - positive or negative - be presented in a paper? Do I need more to present more than odds ratios with associated confidence intervals and p-values?

2) Since the event rate is low and sample size relatively small (and thus underpowered logistic regression), is cross-validation applicable? It seemed like it might give misleading results. If I were doing machine learning/prediction, I'd re-sample to balance (oversample or undersample with something like SMOTE), but I'm also not interested at all in accuracy or other metrics like F1, just the parameter estimates.

In general, what are my options for assessing the results of logistic regression given that I am not interested in prediction?

Improve this question
$\endgroup$
4
  • $\begingroup$ Look at some of the places you might want to publish in. Take reports from them as a guide, not quite a standard since you want to be better than other reports. You should also report some summary information about the total regression, eg, the overall likelihood. You should have some additional analyses that help convince you of your results and might be needed if you are asked for more information. $\endgroup$
    – David Smith
    Commented Jun 1, 2017 at 22:20
  • $\begingroup$ I wouldn't try to use something like SMOTE. If you're really concerned about imbalance, you can bootstrap your parameters. But that sounds more than what's needed for this project. Just present your estimates as they are and speak of the limitations. Don't try to solve everything. $\endgroup$
    – Jon
    Commented Jun 6, 2017 at 20:31
  • $\begingroup$ 8% out of 300 observation is about... 24 samples. 24 samples for four relevant features seems very hopeful even to just "elucidate potential for future research". As suggested by Jon I would also stay away from SMOTE. Bootstrapping is probably the way to go on this. You do not mentioned the distribution of the four relevant features in the sample and that is also important. $\endgroup$
    – usεr11852
    Commented Jun 6, 2017 at 21:02
  • $\begingroup$ Thank you for your comments. The four features are binary. $\endgroup$
    – sharper_image
    Commented Jun 6, 2017 at 22:32

2 Answers 2

Reset to default
3
+50
$\begingroup$

As you are not interested in using your model estimates to make predictions, I don't thin cross-validation is relevant in your case (You don't really seem to care about data over-fitting).

I think you've already pointed out the major limitations of the data - Underpower means that some your results are likely to suffer from type II error (i.e., not rejecting the hypothesis of coeff nullity while you should).

Model log-likelihood in itself is not very informative, except if you use it to compare diff model specifications. For example, you could compare the perf of your "target" model with an empty model (i.e., intercept only), it would tell you something about explanatory power of your independent variables.

You could also look at alternative to R-squared in context of logistic regression (e.g., McFadden Pseudo R2 => McFadden's Pseudo-R2 Interpretation).

You could also compute the % of correctly predicted events (0/1) using your model estimates and compare it to pure chance level (~50%) - It would also tell you something "interesting" about your model.

Improve this answer
$\endgroup$
1
  • $\begingroup$ It should be noted that cross-validation is useful for more than inspecting overfitting in a prediction model. Cross validation can be useful in detecting influential outliers. $\endgroup$
    – Jon
    Commented Jun 7, 2017 at 17:19
0
$\begingroup$

This looks like a situation where going Bayesian makes sense. You have very few positive cases and if you're only interested in the parameter estimates to suggest some directions for future work, a full presentation of the uncertainty associated with those estimates is going to be important, IMO. You might also consider a little bit of regularisation with e.g. a Cauchy prior on the non-intercept parameters.

rstanarm will do Bayesian LR for you in one line of code and it will probably take less than a second to run for this example. If you want some help with it, include your data somewhere and I'll have a look.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.