I've got a study which is kinda messed up by the design...Turns out I ended up with about 50 patients of which 80% have the outcome and 20% don't (binary outcome).

I've been turning in my bed for the last month trying to figure out what to do with this. The only real answer is "don't" but I have to deliver something in the next days as it's part of a master's thesis.

I have about 10 predictors of interest for the outcome (various variable types) and the research question is whether any of these predictors can predict the outcome. Some already have an established correlation in the literature while some are original hypothesis. Several are significant with a univariate logistic regression, but it doesn't look good in any other way than the actual p-value.

Now obviously I'm not going to be able to answer this research question sufficiently but if you were in my shoes, what sort of statistical analysis would you perform to relay to your supervisors?

EDIT: Link to my (anonymized) data (CSV): https://gofile.io/?c=vwH9PS

  • $\begingroup$ What goes wrong if you have adequate power to get significant p-values? $\endgroup$ – Dave Jan 11 '20 at 21:56
  • $\begingroup$ See for yourself. Edited to include data. Outcome variable is labelled outcome. The rest is predictors. $\endgroup$ – Paze Jan 11 '20 at 22:06
  • $\begingroup$ I think I'd go for a form or regularized logistic regression such as elastic net. But as you rightly acknowledge: It's hard to learn much from small samples. $\endgroup$ – COOLSerdash Jan 11 '20 at 22:18
  • $\begingroup$ I've been working with Lasso, would that be the same thing? $\endgroup$ – Paze Jan 11 '20 at 22:23
  • $\begingroup$ Also which of these variables would you include in which bracket of the regularization? I'm interested in all the variables, which makes it hard to choose which go in "interested" and which go in the "picks from" column. $\endgroup$ – Paze Jan 11 '20 at 22:26

For biomedical studies, a general rule of thumb to avoid overfitting in an unpenalized logistic regression model is to have on the order of 10-20 minority-class cases per evaluated predictor. You have about 10 cases in the minority class, so without penalization you should only be evaluating 1 predictor. That predictor would need to be pre-selected based on your knowledge of the subject matter, as using the data to identify the predictor invalidates the assumptions needed to calculate p-values and confidence intervals.

If you did multiple association tests of outcome against each predictor as you propose you would at least have to correct for multiple comparisons and you would not be able to control for the values of the other predictors.

LASSO tends to return a number of predictors similar to the number that would be allowed under the rule of thumb in the first paragraph: so maybe only 1 or 2 in this case.

Logistic ridge regression (L2 penalty) might be the best way to start working with this small data set. All of your predictors would be included in the model, but their coefficients would be heavily penalized to avoid overfitting.

  • $\begingroup$ To be able to correctly understand your answer, which thus far is quite illuminating, I'd like to ask what you mean by the nomenclature "minority-class case"? I've heard the rule of thumb being 10 independent variables per dependent variable but I've never heard of "minority-class cases" and I want to make sure we are on the same page. $\endgroup$ – Paze Jan 11 '20 at 23:03
  • 1
    $\begingroup$ @Paze in logistic regression I use that to mean the class with the smallest number of cases. So it isn’t the 50 total cases that matters. With 40 in one class and only 10 in the other, the rule of thumb would be based on the 10 cases in the smaller class. $\endgroup$ – EdM Jan 12 '20 at 3:15
  • $\begingroup$ Thank you. Is there a similar rule for penalized regressions or can you simply throw as much as you want at them? $\endgroup$ – Paze Jan 12 '20 at 10:10
  • 1
    $\begingroup$ @Paze it’s generally good to start with your understanding of the subject matter rather than just throwing everything blindly at penalized regression software. That said, if predictors are linearly related to outcome then penalized regressions will emphasize the strongest associations in your data sample regardless of how many predictors you include. A risk in throwing everything in is finding spurious associations due to peculiarities of your data sample. A risk of not throwing everything in is missing a true, novel finding. As always: tradeoffs of Type I and Type II errors. $\endgroup$ – EdM Jan 12 '20 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.