3
$\begingroup$

The data for this example can be retrieved here so that you can reproduce these estimates. It is the low birth weight dataset- http://www.umass.edu/statdata/statdata/data/

There are 59 1's and 130 0's for the outcome variable.

I have a sample size of 189. And I run a logistic regression analysis and get these results:

low<-read.delim(file=file.choose(),header=TRUE)
low<-within(low,{
RACE<-factor(RACE,levels=c(1:3),labels=c("White","Black","Hispanic"))})
low.out<-glm(LOW~AGE+FTV+RACE+LWT,data=low,family=binomial)
summary(low.out)


              Estimate      Std.Error    z value      Pr(>|z|)
Intercept     1.295         1.071        1.209        0.227
AGE           -0.024        0.034        -0.706       0.480
FTV           -0.049        0.167        -0.295       0.768
RACE Black     1.004        0.498        2.016        0.044*
RACE Hispanic  0.433        0.362        1.196        0.232
LWT            -0.014       0.007        -2.178       0.029*

So If I wanted to know what the probability was of a black women having a low birth weight baby and she is 30 years old and her weight (in pounds) at the last menstrual period was 108 and she had 1 physician visit during the first trimester, I would calculate the probability as follows. First,

$$1.295-0.024(30)-0.049(1)+1.004(1)+0.433(0)-0.014(108)=0.018.$$

Then, as a probability, $\exp(0.018)/(1+\exp(0.018))*100=50.45\%$.

If the data says the probability that this person will have a low birth weight baby is 50.45%, somebody might question this and say that the sample is only 189.

I only have a sample of 189 and let's say I can't get any more data, How do I convince the layperson that the results/estimates are robust?

Could you do a bootstrapping perhaps? because If I understand correctly, you could resample repeatedly and randomly from the sample like 10 000 times and calculate standard errors and confidence intervals of the regression coefficents (which would make one more confident in the estimates and results). Thereafter, you could get the predicted probabilities and the 95% confidence intervals? If this is the case, how would I do the bootstrapping in R for this example?

$\endgroup$
4
$\begingroup$

Compared to many other situations such as in NLP (natural language processing), you have 189 samples and 4 features which are not bad.

Besides, the example you gave is a typical example (you should have seen many similar examples) for your samples. That is an intuitive reason why your prediction should not be "so wrong".

I think the bootstrapping won't help here in this case because you don't introduce any new information into the samples. If you are able to introduce some more information and then create "virtual samples", this would be helpful. However, it seems dangerous here unless there are already proved medical evidence to justify this approach.

Finally, I have the impression that here it is the variable selection procedure which makes your regression model good or bad. Carefully choose the right variable to integrate into the regression model can make the results more convincing. (Perhaps it is something you have already done as your model has fewer variables than the original file).

$\endgroup$
  • $\begingroup$ Thanks, but without having any more information, doesn't the addition of a virtual sample count as some sort of data fabrication? "If you are able to introduce some more information and then create "virtual samples", this would be helpful."... Please possibly see my question too: stats.stackexchange.com/questions/398491/… $\endgroup$ – Vic Mar 21 at 8:05
4
$\begingroup$

To support your model you could quantify a number of issues:

  1. You have 59 events and ? 5 degrees of freedom. ~11 events per variable (EPV). 10-20 is the rule of thumb. This isn't terrible. Not great. Commonly used metric and worth documenting.
  2. A pre-specified model with have the greatest validity with this small sample side. You'll have to convince an audience that you didn't do any variable selection or be explicit about variable/model choice. (And if you did why this didn't lead to overfitting)
  3. The data can reasonable assumed to be representative of future data.
  4. You can bootstrap (http://www.ncbi.nlm.nih.gov/pubmed/11470385). But in this setting you often bootstrap to validate model building process rather than estimates - to provide a estimation of model optimism. I believe linked article is open access, but google "steyerberg internal validation" will work just as well.
$\endgroup$
  • $\begingroup$ Hi there Charles, thanks for your comments. So does that mean if I had 100 events and 5 degrees of freedom, it's 20 events per variable? I'm new to this bootstrapping concept, and I was spurred on by this after reading this post on the forum: stats.stackexchange.com/questions/59829/… even using the code there, I still didn't know how I would use the bias corrected confidence intervals or even if they were right or not and how to say whether my probability of 50.45% would change for the black woman, as in my original post. $\endgroup$ – user3497385 Oct 28 '14 at 19:16
  • $\begingroup$ I thought maybe I could somehow use the bias corrected confidence intervals for the estimates in some way to create a new probability value for the black woman and say, well, I resampled 10 000 times from the sample (with replacement), and I'm pretty confident that the probability of this woman having a low birth baby is around 50%. That was kind of what I was hoping for with bootstrapping :-) $\endgroup$ – user3497385 Oct 28 '14 at 19:19
  • $\begingroup$ @user3497385 (1) yes, 100 events/5 = 20 epv. One chooses the least frequent event (in your case 1) to base the calculation. (2) don't trust me on this - I'm not sure. But usually in terms of prediction and small samples I usually think the coefficients and SE are biased. Thus one often talks of shrinkage factors to correct for this (usually the coefficients) Again Steyeberg: yaroslavvb.com/papers/steyerberg-application.pdf for bootstrapping CE I leave for when I think the assumptions parametric of my model might not hold so they might be biased. $\endgroup$ – charles Oct 28 '14 at 23:34
  • $\begingroup$ but this is very much from the world of prediction, if prediction isn't the main goal - shrinkage is very rarely done and boostrapping CE might well be reasonable. $\endgroup$ – charles Oct 29 '14 at 0:08
  • $\begingroup$ Is there a way to increase the test power via bootstrapping? Please see my similar question, and possibly let me know your opinion. Thanks. stats.stackexchange.com/questions/398491/… $\endgroup$ – Vic Mar 21 at 8:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.