First I’ll explain the data and then the problem.


Let’s say that we need to build a GLM model to predict the income of $N$ individuals. The income variable is positive and skewed to the right, thus we assume it follows Gamma distribution. We have three continuous predictors, $X_1$, $X_2$, and $X_3$, with $X_i \in [0, 100]$, for $i=1,2,3$. Also, we assume that they are not correlated with each other.

Also, let’s assume that $X_1$ and $X_2$ are negatively correlated with income, while $X_3$ is positively correlated. So after building the GLM model (gamma with log-link), $X_1$ and $X_2$ will have negative coefficients while $X_3$ has a positive coefficient.

Also, for illustration, let’s assume that $\hat{\beta}_0 = 0.5$, $\hat{\beta}_1 = -0.1$, $\hat{\beta}_2 = -0.2$, and $\hat{\beta}_3 = 0.3$,


Let’s say that both $X_1$ and $X_2$ are large values (say, 90 and 90) and also $X_3$ is large (equals 100), then the fitted income value will be lowered because $X_1$ and $X_2$ have large values. What I want is if $X_3$ is large I need to give it more weight and kind of ignore $X_1$ and $X_2$ contributions. I feel that GLM is not appropriate here. Do you have suggestions of using other models, perhaps non-linear, and if so, what the structure would be?

  • 1
    $\begingroup$ If we are to assume that the model coefficients represent the "weights" of which you speak here, then no the GLM cannot do what you want, such as kind-of ignoring two other variables' contributions to the estimated effects: the GLM is a maximum likelihood procedure, and hence does not conform to beliefs (outside of the set-up of the model itself). It sounds like you wish to estimate a Bayesian GLM with an informative prior. $\endgroup$
    – AdamO
    Dec 21 '17 at 17:40
  • 1
    $\begingroup$ 1. Why would you want to give $X_3$ a higher coefficient (in effect) if it's large? 2. It sounds like what you want is a nonlinear model, perhaps, in which case a GAM might be more appropriate - especially one that can estimate some interactions. $\endgroup$
    – jbowman
    Dec 21 '17 at 17:49
  • 1
    $\begingroup$ If you derive suitable regressors and the data support your suspected behaviour, GLM might be able to do the job. $\endgroup$
    – Michael M
    Dec 21 '17 at 18:37
  • 1
    $\begingroup$ Two points: 1) from my experience, the Gamma distribution has underestimated the tail probabilities for incomes in the United States. I had to used a mixture distribution to fit the tail correctly. 2) if you need a larger weight on $X_3$ when it is larger, why not add $X^2_3$ (or some other power, like 1.5) to the model? $\endgroup$ Dec 21 '17 at 19:38
  • $\begingroup$ @AdamO That's exactly what I'm thinking. GLM is what is given in the coefficients and cannot be changed. In the Bayesian setup, how this would work? how I can inform the model that if X1, X2, and X3 are large then give X1 and X2 small weights? $\endgroup$
    – user9292
    Dec 21 '17 at 19:40

What you are describing here is a non-linearity. From a modeling perspective, most non-linearities can be accommodated by creating a clever set of derived variables. The model which only adjusts for $X_1$, $X_2$, $X_3$ as covariates is not correct. Therefore this is not so much a limitation of the GLM as it is one of specifying the right mean model.

For this answer, a caveat: large is not well-defined. Is it 40? 80? I am assuming we know what large is, it is a clear break-point, and once we exceed it, linear effects in $X_1, X_2, X_3$ are otherwise acceptable.

You can create the indicator variable: x3large as 0 if $X_3$ is small and 1 if $X_3$ is large. Take the product of this indicator variable with the other 3 $X_1, X_2, X_3$. The numerator degrees of freedom for the model now goes from 4 to 7. The interaction terms' coefficients represent a difference in coefficient slopes when $X_3$ is large. When the interaction term is of opposite sign but less than the original coefficient, it represents a decrease in effect. If they are of the same sign, the effect is larger in that large domain.

Per your scientific description of the trend: if the data resemble the process you have described, maximizing the likelihood is all you need to do. The interaction with x3large and x1, x2 will be negative to attenuate their effect. The interaction with x3large and x3 will be positive to "ramp up" it's effect.

Now I advise stopping here. If the optimal fit to the data is not as you describe, this is an interesting result. We assumed a trend and results were found otherwise.

It is possible however to fit models which are constrained to achieve effects only as you describe. The first method is constrained optimization. You can add a lagrange multiplier to the likelihood to offset the cases when the interaction terms are not of the desired sign. Similarly, you can conduct a Bayesian analysis with 0 prior probability on those values. The predictions from the constrained models could be compared to the unrestricted, full model to make qualitative inference on the validity of either approach.

  • $\begingroup$ Thank you for the tips. Do you have a reference with examples for the constrained optimization method? $\endgroup$
    – user9292
    Dec 22 '17 at 18:52
  • $\begingroup$ See ?stats::constrOptim and search site posts. The relevant text: Numerical Analysis for Statisticians by Lange for more details. $\endgroup$
    – AdamO
    Dec 22 '17 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.