I am trying to better understand the difference between the Average of a Variable vs. the "Average Effect' of the Same Variable in a Statistical Model.

To illustrate my example, I use the R programming language and the Iris Dataset.

1) (Assuming that the data comes from a Normal Distribution) It is generally straightforward to calculate the mean of a random variable (using the OLS or MLE estimator):


#remove one of the species in this example
iris_mod = iris[which(iris$Species != "setosa"), ]


   Sepal.Length Sepal.Width Petal.Length Petal.Width    Species
51          7.0         3.2          4.7         1.4 versicolor
52          6.4         3.2          4.5         1.5 versicolor
53          6.9         3.1          4.9         1.5 versicolor
54          5.5         2.3          4.0         1.3 versicolor
55          6.5         2.8          4.6         1.5 versicolor
56          5.7         2.8          4.5         1.3 versicolor

[1] 6.262

[1] 2.872

[1] 4.906

[1] 1.676


meltData <- melt(iris_mod)

p<-ggplot(meltData, aes(x=variable, y=value, fill=variable)) +
  geom_boxplot() + ggtitle("Distribution of Variables")

enter image description here

2) Now, suppose we want to fit a (binomial) logistic regression model to this data:

 glm(formula = Species ~ ., family = "binomial", data = iris_mod)
 Deviance Residuals: 
      Min        1Q    Median        3Q       Max  
 -2.01105  -0.00541  -0.00001   0.00677   1.78065  
              Estimate Std. Error z value Pr(>|z|)  
 (Intercept)   -42.638     25.707  -1.659   0.0972 .
 Sepal.Length   -2.465      2.394  -1.030   0.3032  
 Sepal.Width    -6.681      4.480  -1.491   0.1359  
 Petal.Length    9.429      4.737   1.991   0.0465 *
 Petal.Width    18.286      9.743   1.877   0.0605 .
 Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
 (Dispersion parameter for binomial family taken to be 1)
     Null deviance: 138.629  on 99  degrees of freedom
 Residual deviance:  11.899  on 95  degrees of freedom
 AIC: 21.899
 Number of Fisher Scoring iterations: 10 

As we can see, the model estimates for the effects of the variables are quite different from the actual averages of the same variables (e.g. certain quantities that correspond to "real" measurements in the natural world have "negative" estimates):

  variable_name actual_variable_average model_estimate
1  Sepal.Length                   6.262         -2.465
2   Sepal.Width                   2.872         -6.681
3  Petal.Length                   4.906          9.429
4   Petal.Width                   1.676         18.286

My Question: When faced with this discrepancy in the real world (i.e. average of the variables in the real world vs. the beta coefficients in a regression model) - how is someone expected to place Bayesian Priors on the parameters of the logistic regression model?

For example, the measurements of the Iris Flowers were recorded in centimeters. Suppose my friend is a botanist and tells me that "iris flowers" in the real world tend to have an average petal length of 5 cm and a standard deviation of 0.5 cm; my friend also tells me that these measurements tend to have a normal distribution as well. I consider this to be valuable information and think that my regression model can benefit from this "prior information". How do I incorporate what my friend told me into the regression model - when I only have prior information only about the "average of the variables in the real world", and not the information about the "average effect of the variable in the model"?

enter image description here

Naturally, my botanist friend only knows about the nature of the well-studied data in the real world - and not the nature of my model parameters. Is this problem actively being studied in the real world? Can someone please provide a comment on this?

Note: Instead of regression models - if you choose to model iris data using a Multivariate Normal Distribution or a Copula Model, I think it will be easier to incorporate real world prior knowledge into the model directly (since the model parameters of a Multivariate Normal Distribution would correspond directly to the known means and variances of the flower measurements)

enter image description here

  • 1
    $\begingroup$ The coefficients in a logistic regression model are log odds ratios... why would you expect them to be equal to the means of the predictors? $\endgroup$
    – Noah
    Nov 21, 2021 at 9:31
  • $\begingroup$ @ noah: thank you for your reply! Of course I don't expect them to be equal (as you pointed out, the model parameters and the means of the variables are two completely different things) - i was just trying to illustrate a point: in the context of choosing priors for bayesian models, we have real world information about the actual variables and not the model parameters. Bayesian models require information about the model parameters : how can we bridge this gap in the real world when creating bayesian models? $\endgroup$
    – stats_noob
    Nov 21, 2021 at 9:35
  • 1
    $\begingroup$ Better change the title of your question, and maybe also the beginning. What you do there seems irrelevant at first sight for what you now say is actually your question. For starters, you are not supposed to use any information in the data for choosing the prior, otherwise it wouldn't be a "prior". Thinking a bit more I understand the relevance, but still maybe start with the question and then illustrate your point. That would make it less hard for the reader. $\endgroup$ Nov 21, 2021 at 11:16
  • $\begingroup$ @ christian henning : thank you for your reply! I will try to make the edits you suggested. Of course you are not supposed to use information in your observed dataset as a prior - that defeats the whole purpose! But in my example, the available information to define the priors comes outside of the dataset (i.e. a field expert on flowers). $\endgroup$
    – stats_noob
    Nov 21, 2021 at 17:12


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.