1
$\begingroup$

My situation:

  • sample size: 116
  • binary outcome (32 events)
  • number predictors: 42 (both continuous and categorical)
  • predictors did not come from the top of my head; their choice was based on the literature.

I have run group lasso logistic regression as explained in suggested in a previous post, aiming to identify the most important variables in explaining my outcome. I am less interested in making predictions to much larger populations - my sample represents roughly 90% of the population I am most interest in.

The lasso analyses resulted in 9 predictors with nonzero coefficients. As recommended in a previous post, I conducted bootstraping to assess reproducibility and Monte Carlo simulation to assess the methods' ability to find the true variables. I got the following results for the bootstrapping:

  • on average, 78% (7 out of 9) of the variables in original model entered the models generated for the 100 bootstrap samples. On the other hand, only 41% (7 out of 17), for the other way around;

  • 2 variables entered the new models more then 95% of times, 3 variables more than 90%, 5 variables more than 85% and 6 variables more than 75%. Those 6 variables are among the 9 included in the original model;

and the following one for the Monte Carlo simulation:

  • 44% (4 out of 9) of predictors composing the original set (assumed as "true") match the ones that entered the models generated in the Monte Carlo simulation, and 44% (4 out of 9) vice-versa.

My overall question is: what would be an appropriate way to interpret those results? Can I say that my results are reproducible to other similar samples and that the method is able to select the true variables to an acceptable level? Also:

  • Could I list all the variables and how frequently they enter the model generated for the bootstrap samples, and then argue that those that enter more frequently are more likely to be significant?

  • Could I say that there is evidence that the original feature set is relevant in explaining the outcome (since ~7 out of the 9 variables entered the bootstrap samples' models on average and considering individual feature inclusion fractions) but prediction would be likely to be enhanced if additional variables are included (since bootstrap sample models tended to include 17 variables on average)?

  • Can anyone suggest some references about the use of bootstrap and Monte Carlo in the context suggested in this post (analysis of zero versus non-zero predictor coefficients rather than estimation of coefficients' standard deviation or the like)?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ How are you going to use these results? It's not really clear that there is typically a true set of predictor variables, just sets of useful ones for particular purposes. Since your work draws on published literature, consider whether interpretation in terms of that literature might be more important. With respect to "zero versus non-zero predictor coefficients," I don't think there is a way to make such a distinction without effectively using information on "coefficients' standard deviation or the like"; the issue is whether 0 can be ruled out as a coefficient, given the variability. $\endgroup$
    – EdM
    Oct 17, 2014 at 20:13
  • $\begingroup$ @EdM: Isn't there always a true set of predictors? As I understand it (I may have got it wrong) they would be the right or significant predictors, considering the whole population you have sampled from. Maybe I have not made it clear enough, I want to explain what is going on in the population that I am studying, what are the factors that are most important in influencing the outcome. I think I see the point of your comment on "zero versus non-zero...", but I got that calculating coefficients standard deviations is problematic for lasso. Does that mean that my analyses are misguided? $\endgroup$
    – Puzzled
    Oct 17, 2014 at 20:33
  • 1
    $\begingroup$ I don't believe there is always a true set of predictors. There may be a set that is statistically significant by some criterion after analysis of a particular sample of the population, but a larger sample would typically make it possible to detect a larger set of "significant," true predictors. As I believe you have been warned previously, with only 32 events you are unlikely to be able to document more than 2 or 3 significant predictors (about 15 events needed per predictor). If you had a larger sample with 320 events, you would have been able to document several more. So which are true? $\endgroup$
    – EdM
    Oct 17, 2014 at 21:22
  • $\begingroup$ @EdM:So, where I refer to true predictors in my question, would it be appropriated to refer to most significant predictors instead? $\endgroup$
    – Puzzled
    Oct 19, 2014 at 14:35
  • 1
    $\begingroup$ You could say that certain predictors were most significantly related to outcome, based on specified criteria, but I caution against inferring causation from such a statement. If this is for a report or publication, also include summaries of your bootstrap and Monte Carlo results so that readers can judge for themselves. $\endgroup$
    – EdM
    Oct 19, 2014 at 17:53

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.