3
$\begingroup$

I've got a study which is kinda messed up by the design...Turns out I ended up with about 50 patients of which 80% have the outcome and 20% don't (binary outcome).

I've been turning in my bed for the last month trying to figure out what to do with this. The only real answer is "don't" but I have to deliver something in the next days as it's part of a master's thesis.

I have about 10 predictors of interest for the outcome (various variable types) and the research question is whether any of these predictors can predict the outcome. Some already have an established correlation in the literature while some are original hypothesis. Several are significant with a univariate logistic regression, but it doesn't look good in any other way than the actual p-value.

Now obviously I'm not going to be able to answer this research question sufficiently but if you were in my shoes, what sort of statistical analysis would you perform to relay to your supervisors?

EDIT: Link to my (anonymized) data (CSV): https://gofile.io/?c=vwH9PS

Improve this question
$\endgroup$
5
  • $\begingroup$ What goes wrong if you have adequate power to get significant p-values? $\endgroup$
    – Dave
    Commented Jan 11, 2020 at 21:56
  • $\begingroup$ See for yourself. Edited to include data. Outcome variable is labelled outcome. The rest is predictors. $\endgroup$
    – Paze
    Commented Jan 11, 2020 at 22:06
  • $\begingroup$ I think I'd go for a form or regularized logistic regression such as elastic net. But as you rightly acknowledge: It's hard to learn much from small samples. $\endgroup$
    – COOLSerdash
    Commented Jan 11, 2020 at 22:18
  • $\begingroup$ I've been working with Lasso, would that be the same thing? $\endgroup$
    – Paze
    Commented Jan 11, 2020 at 22:23
  • $\begingroup$ Also which of these variables would you include in which bracket of the regularization? I'm interested in all the variables, which makes it hard to choose which go in "interested" and which go in the "picks from" column. $\endgroup$
    – Paze
    Commented Jan 11, 2020 at 22:26

1 Answer 1

Reset to default
5
$\begingroup$

For biomedical studies, a general rule of thumb to avoid overfitting in an unpenalized logistic regression model is to have on the order of 10-20 minority-class cases per evaluated predictor. You have about 10 cases in the minority class, so without penalization you should only be evaluating 1 predictor. That predictor would need to be pre-selected based on your knowledge of the subject matter, as using the data to identify the predictor invalidates the assumptions needed to calculate p-values and confidence intervals.

If you did multiple association tests of outcome against each predictor as you propose you would at least have to correct for multiple comparisons and you would not be able to control for the values of the other predictors.

LASSO tends to return a number of predictors similar to the number that would be allowed under the rule of thumb in the first paragraph: so maybe only 1 or 2 in this case.

Logistic ridge regression (L2 penalty) might be the best way to start working with this small data set. All of your predictors would be included in the model, but their coefficients would be heavily penalized to avoid overfitting.

Improve this answer
$\endgroup$
4
  • $\begingroup$ To be able to correctly understand your answer, which thus far is quite illuminating, I'd like to ask what you mean by the nomenclature "minority-class case"? I've heard the rule of thumb being 10 independent variables per dependent variable but I've never heard of "minority-class cases" and I want to make sure we are on the same page. $\endgroup$
    – Paze
    Commented Jan 11, 2020 at 23:03
  • 1
    $\begingroup$ @Paze in logistic regression I use that to mean the class with the smallest number of cases. So it isn’t the 50 total cases that matters. With 40 in one class and only 10 in the other, the rule of thumb would be based on the 10 cases in the smaller class. $\endgroup$
    – EdM
    Commented Jan 12, 2020 at 3:15
  • $\begingroup$ Thank you. Is there a similar rule for penalized regressions or can you simply throw as much as you want at them? $\endgroup$
    – Paze
    Commented Jan 12, 2020 at 10:10
  • 1
    $\begingroup$ @Paze it’s generally good to start with your understanding of the subject matter rather than just throwing everything blindly at penalized regression software. That said, if predictors are linearly related to outcome then penalized regressions will emphasize the strongest associations in your data sample regardless of how many predictors you include. A risk in throwing everything in is finding spurious associations due to peculiarities of your data sample. A risk of not throwing everything in is missing a true, novel finding. As always: tradeoffs of Type I and Type II errors. $\endgroup$
    – EdM
    Commented Jan 12, 2020 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.