2
$\begingroup$

I have multiple logistic model outputs with the same predictors. The dependent response is very similar changing slightly randomly.

I would like to take an average of the coefficients of all models. But I'm concerned that some coefficients may be large with non-significant p values and other differences. How can I get an overall average measure of effect for each predictor while considering the standard error/statistic/p-value aspects?

I thought of penalizing the coefficient each time it is not significant at a predetermined threshold. Something like:

#Edit:: This has been concluded to be the wrong way
mdl_list <- lapply(predictors, function(y) {
              tbl <- broom::tidy(glm(y, x))
              tbl$estimate <- ifelse(tbl$p.value > .05, 0, tbl$estimate)
              tbl
              }
            )

If an alternative approach to average effect exists, please point me in the right direction.

Edit

I have tried the meta-analysis in the metafor package. I even emailed the authors, they said that if this is not for summarizing multiple studies, metafor will not work.

Example: The b variables are response, the q variables are predictors.

#Data
set.seed(143)
b <- replicate(3, rbinom(8, 1, .8))
q <- replicate(4, sample(1:5, 8, TRUE))
df <- data.frame(b,q)
names(df) <- c(paste0("b",1:3), paste0("q",1:4))
#   b1 b2 b3 q1 q2 q3 q4
# 1  0  1  1  2  1  2  1
# 2  1  1  1  5  1  1  4
# 3  1  1  1  5  5  1  5
# 4  1  1  1  4  5  3  1
# 5  1  0  0  2  5  4  1
# 6  0  1  1  4  5  2  4
# 7  1  0  0  1  2  3  5
# 8  1  0  0  1  1  5  4

mdl1 <- glm(b1 ~ q1 + q2 + q3 + q4, data=df, family=binomial())
mdl2 <- glm(b2 ~ q1 + q2 + q3 + q4, data=df, family=binomial())
mdl3 <- glm(b3 ~ q1 + q2 + q3 + q4, data=df, family=binomial())

Edit 2

I may have to combine the different brands into one y variable and repeat the question responses:

y <- with(df, c(b1, b2, b3))
newdf <- df[rep(1:8,3),-(1:3)]
newdf$y <- y
head(newdf, 10)
#     q1 q2 q3 q4 y
# 1    2  1  2  1 0
# 2    5  1  1  4 1
# 3    5  5  1  5 1
# 4    4  5  3  1 1
# 5    2  5  4  1 1
# 6    4  5  2  4 0
# 7    1  2  3  5 1
# 8    1  1  5  4 1
# 1.1  2  1  2  1 1
# 2.1  5  1  1  4 1

mdl4 <- glm(y ~ q1 + q2 + q3 + q4, data=newdf, family=binomial())

The predictors q are questions like on a scale of 1-5 how much do you agree with this statement, "Store brands are just as good as name brands", "Organic foods are worth their price", "I look for discounts when I go to the supermarket".

The response variables b are questions like "Do you eat Dannon yogurt?", "Do you eat Campbell's soup?", "Do you eat at McDonald's?".

The hypothesis is that there are some attitudes questions that are better than others at predicting brand usage. If I run a logistic regression:

$ShopMcD's = \beta_0 + \beta_1 Attitude_1 + ... + \beta_n Attitude_n$

I can see how much each question affects the log odds ratio of shopping at McDonald's. The higher ups want to see this continued for all of the food brands to see the effect each question has on brand usage overall.

An example project delivery statement would be:

"We find that attitudes questions 2, 4, and 6 have the least effect on average over all other questions. These questions are the least predictive of food brand usage compared to others and are candidates for removal from the survey."

$\endgroup$
16
  • 5
    $\begingroup$ The best improvement that comes to my mind is not doing what you want to do... $\endgroup$
    – Tim
    Sep 22 '16 at 19:51
  • $\begingroup$ Have you thought of meta-analysis which synthesises estimates from different studies? $\endgroup$
    – mdewey
    Sep 22 '16 at 20:23
  • 1
    $\begingroup$ @PierreLafortune I had assumed that you had multiple studies as well. Now that I look at your edit, I see that your regressions are using the same explanatory variables q1 through q4. I no longer understand what your intention is, and I think that you're heading down a dead-end. If you somehow want to "average" predictions of b1, b2, and b3, perhaps you want to do so directly? e.g. define "y = b1 OR b2 OR b3" or "y = b1 AND b2 AND b3," whichever one is closer to you intention, and perform y~q1+q2+q3+q4. $\endgroup$
    – jwimberley
    Sep 26 '16 at 21:21
  • 1
    $\begingroup$ @PierreLafortune Yes, intentionally different, because your current test makes no sense. If b1, b2, and b3 are different things, what is the average of the regression coefficients supposed to mean? They'll give you some "average" probability for the 3 regressions, e.g. approximately $(p(b1) + p(b2) + p(b3))/3$, which is like $p(b1 \wedge b2 \wedge b3)$ except that it is ill defined. $\endgroup$
    – jwimberley
    Sep 26 '16 at 21:28
  • 2
    $\begingroup$ It isn't at all clear what an "average effect" is supposed to mean. Your proposal to concatenate the datasets suggests that b1, b2, and b3 are independent observations of the same variable rather than being different variables. What are these variables, really, and what are you trying to learn about them? $\endgroup$
    – whuber
    Sep 26 '16 at 22:05
3
$\begingroup$

Since you use R you might like to look at some of the packages listed in the CRAN TaskView (disclaimer, I maintain it). My personal preference is the metafor package and its author has a wealth of explanatory material on the package website. In the unlikely event that you cannot find the answer to your problems there ask again.

$\endgroup$
3
  • $\begingroup$ The metafor package will not work for my case. I have explored the package documents and emailed the author of the package. They say it will not work. $\endgroup$
    – Pierre L
    Sep 26 '16 at 21:10
  • $\begingroup$ Yes, I had understood your original question to be about multiple datasets. Apologies. $\endgroup$
    – mdewey
    Sep 27 '16 at 12:14
  • $\begingroup$ Thank you, it's the same study. I'm looking for the central tendency of the predictors over dependent variables I care about. $\endgroup$
    – Pierre L
    Sep 27 '16 at 12:16
1
$\begingroup$

Your idea of averaging regression coefficients is ill-defined, and your new idea in the second edit is not good because it repeats datapoints in the dataset and will have underestimated uncertainties (edit: unless b1, b2, and b3 are supposed to be independent observations at the same q values, as @whuber ponders, though I get the impression otherwise). Your problem is that you don't know what effect you are trying to measure. Your question contains four explanatory variables q1, q2, q3, and q4, with three simultaneous boolean outcomes b1, b2, and b3. The only valid statistical questions you can answer with binomial regression are of the form "what is the probability of y as a function of q1 through q4, where y is a well-defined event or outcome that is related to b1, b2, and b3?" For example, y could be b1 || b2 || b3, b1 && b2 && b3, b1 && (b2 || !b3), etc. Performing different regressions for the outcomes b1, b2, and b3 using the same input data is fine, but there is no meaningful way to combine them. If you're not trying to the predict the probability of some y that can be written with b1, b2, and b3 using boolean algebra, then what are you trying to predict? Until you can answer this you have a answer in search of a question.

$\endgroup$
5
  • $\begingroup$ I am trying to predict each brand against the questions individually then summarise the findings. Instead of saying for brand1 the coefficients are x, for brand2 the coefficients are this, and continue on for one hundred plus brands. This is a flood of data. The bosses are asking, "How predictive are the questions of brand usage, period?" Not just one brand, or two, or a combination of two. Overall, how good are the questions at predicting usage, and let's remove the questions that are the worst at predicting brand usage from future surveys. $\endgroup$
    – Pierre L
    Sep 27 '16 at 4:00
  • $\begingroup$ @PierreLafortune You need to decide exactly how to quantify "Overall". If you just want evidence that any of the logistic regressions shows a relationship, you should look at their individual significance tests. Or, you should determine if one my suggestions in the answer corresponds better to "Overall." Note that perhaps $b1 \sim 0.4 - 0.1q1$ and $b2 \sim 0.4 +0.1 q1$ so that if you average the coefficients you'll get something close to zero even if they are separately statistically significant. $\endgroup$
    – jwimberley
    Sep 27 '16 at 12:11
  • $\begingroup$ It would be important to know that two models returned conflicting estimations. I think it would indicate a predictor that does not have a uniform (dependable) effect on the predictors we care about. That's evidence that averaging is getting closer to the central tendency of that variable. Do you agree? $\endgroup$
    – Pierre L
    Sep 27 '16 at 12:14
  • $\begingroup$ @PierreLafortune No. I agree that you can compare the different regression coefficients, to say "q1's effect on b1 is different than its effect on b2", or say that "q1's effect is statistically strong in x out of y cases," but I do not think that averaging them is well-defined or gives a central tendency. I still think that defining y = b1 || b2 || b3 is really what you want to be doing, if you'll think about it. $\endgroup$
    – jwimberley
    Sep 27 '16 at 13:10
  • $\begingroup$ I think your suggestion of looking at the significance tests is useful. I started out the project using it but gave up thinking it wasn't good enough. $\endgroup$
    – Pierre L
    Sep 27 '16 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.