14
$\begingroup$

This is a follow up question from the one I asked a couple of days ago. I feel it puts a different slant on the issue, so listed a new question.

The question is: can I compare the magnitude of coefficients across models with different dependent variables? For example, on a single sample say I want to know whether the economy is a stronger predictor of votes in the House of Representatives or for President. In this case, my two dependent variables would be the vote in the House (coded 1 for Democrat and 0 for Republican) and vote for President (1 for Democrat and 0 for Republican) and my independent variable is the economy. I'd expect a statistically significant result in both offices, but how do I assess whether it has a 'bigger' effect in one more than the other? This might not be a particularly interesting example, but i'm curious about whether there is a way to compare. I know one can't just look at the 'size' of the coefficient. So, is comparing coefficients on models with different dependent variables possible? And, if so, how can it be done?

If any of this doesn't make sense, let me know. All advice and comments are appreciated.

$\endgroup$
4
  • 2
    $\begingroup$ How do you know one can't just look at the 'size' of the coefficient? $\endgroup$
    – onestop
    Mar 26, 2011 at 9:40
  • $\begingroup$ I merged your two accounts. You'll still need to register, as indicated on the FAQ. (@onestop Thx for pointing to the duplicate.) $\endgroup$
    – chl
    Mar 26, 2011 at 12:33
  • $\begingroup$ I assumed that I couldn't compare 'effect' of predictors across models by looking at the coefficients from the answers to my previous question. Are things different for my example above? $\endgroup$
    – Ejs
    Mar 27, 2011 at 4:24
  • 2
    $\begingroup$ Starting a bounty - seems like an important question with three very different answers, none of which has a single vote. We can do better. Andy W's paper link on this related question seems pertinent. $\endgroup$ Apr 21, 2011 at 16:09

4 Answers 4

4
$\begingroup$

The short answer is "yes you can" - but you should compare the Maximum Likelihood Estimates (MLEs) of the "big model" with all co variates in either model fitted to both.

This is a "quasi-formal" way to get probability theory to answer your question

In the example, $Y_{1}$ and $Y_{2}$ are the same type of variables (fractions/percentages) so they are comparable. I will assume that you fit the same model to both. So we have two models:

$$M_{1}:Y_{1i}\sim Bin(n_{1i},p_{1i})$$ $$log\left(\frac{p_{1i}}{1-p_{1i}}\right)=\alpha_{1}+\beta_{1}X_{i}$$ $$M_{2}:Y_{2i}\sim Bin(n_{2i},p_{2i})$$ $$log\left(\frac{p_{2i}}{1-p_{2i}}\right)=\alpha_{2}+\beta_{2}X_{i}$$

So you have the hypothesis you want to assess:

$$H_{0}:\beta_{1}>\beta_{2}$$

And you have some data $\{Y_{1i},Y_{2i},X_{i}\}_{i=1}^{n}$, and some prior information (such as the use of logistic model). So you calculate the probability:

$$P=Pr(H_0|\{Y_{1i},Y_{2i},X_{i}\}_{i=1}^{n},I)$$

Now $H_0$ doesn't depend on the actual value of any of the regression parameters, so they must have be removed by marginalising.

$$P=\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} Pr(H_0,\alpha_{1},\alpha_{2},\beta_{1},\beta_{2}|\{Y_{1i},Y_{2i},X_{i}\}_{i=1}^{n},I) d\alpha_{1}d\alpha_{2}d\beta_{1}d\beta_{2}$$

The hypothesis simply restricts the range of integration, so we have:

$$P=\int_{-\infty}^{\infty} \int_{\beta_{2}}^{\infty} \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} Pr(\alpha_{1},\alpha_{2},\beta_{1},\beta_{2}|\{Y_{1i},Y_{2i},X_{i}\}_{i=1}^{n},I) d\alpha_{1}d\alpha_{2}d\beta_{1}d\beta_{2}$$

Because the probability is conditional on the data, it will factor into the two separate posteriors for each model

$$Pr(\alpha_{1},\beta_{1}|\{Y_{1i},X_{i},Y_{2i}\}_{i=1}^{n},I)Pr(\alpha_{2},\beta_{2}|\{Y_{2i},X_{i},Y_{1i}\}_{i=1}^{n},I)$$

Now because there is no direct links between $Y_{1i}$ and $\alpha_{2},\beta_{2}$, only indirect links through $X_{i}$, which is known, it will drop out of the conditioning in the second posterior. same for $Y_{2i}$ in the first posterior.

From standard logistic regression theory, and assuming uniform prior probabilities, the posterior for the parameters is approximately bi-variate normal with mean equal to the MLEs, and variance equal to the information matrix, denoted by $V_{1}$ and $V_{2}$ - which do not depend on the parameters, only the MLEs. so you have straight-forward normal integrals with known variance matrix. $\alpha_{j}$ marginalises out with no contribution (as would any other "common variable") and we are left with the usual result (I can post the details of the derivation if you want, but its pretty "standard" stuff):

$$P=\Phi\left(\frac{\hat{\beta}_{2,MLE}-\hat{\beta}_{1,MLE}}{\sqrt{V_{1:\beta,\beta}+V_{2:\beta,\beta}}}\right) $$

Where $\Phi()$ is just the standard normal CDF. This is the usual comparison of normal means test. But note that this approach requires the use of the same set of regression variables in each. In the multivariate case with many predictors, if you have different regression variables, the integrals will become effectively equal to the above test, but from the MLEs of the two betas from the "big model" which includes all covariates from both models.

$\endgroup$
3
+50
$\begingroup$

Why not? The models are estimating how much 1 unit of change in any model predictor will influence the probability of "1" for the outcome variable. I'll assume the models are the same-- that they have the same predictors in them. The most informative way to compare the relative magnitudes of any given predictor in the 2 models is to use the models to calculate (either deterministically or better by simulation) how much some meaningful increment of change (e.g., +/- 1 SD) in the predictor affects the probabilities of the respective outcome variables--& compare them! You'll want to determine confidence intervals for the two estimates as well as so you can satisfy yourself that the difference is "significant," practically & statistically.

$\endgroup$
2
  • $\begingroup$ Thanks dmk8, very useful. Some follow-up points/questions: is this what is often meant when referring to varying the variable of interest (the economy from bad to good for example) while holding all control variables at their means? What do you mean by deterministically? How do I determine the confidence intervals around the probabilities? $\endgroup$
    – Ejs
    Apr 29, 2011 at 4:06
  • 2
    $\begingroup$ Consult the King. He will not disappoint. King, G., Tomz, M., & Wittenberg., J. (2000). Making the Most of Statistical Analyses: Improving Interpretation and Presentation. Am. J. Pol. Sci, 44(2), 347-361. $\endgroup$
    – dmk38
    May 13, 2011 at 22:45
2
$\begingroup$

I assume that by "my independent variable is the economy" you're using shorthand for some specific predictor.

At one level, I see nothing wrong with making a statement such as

X predicts Y1 with an odds ratio of _ and a 95% confidence interval of [ _ , _ ] while X predicts Y2 with an odds ratio of _ and a 95% confidence interval of [ _ , _ ].

@dmk38's recent suggestions look very helpful in this regard.

You might also want to standardize the coefficients to facilitate comparison.

At another level, beware of taking inferential statistics (standard errors, p-values, CIs) literally when your sample constitutes a nonrandom sample of the population of years to which you might want to generalize.

$\endgroup$
2
  • $\begingroup$ Yes, 'the economy' is shorthand for perceptions of national economic conditions. Does the same advice apply when other predictors (controls) are included in the model? $\endgroup$
    – Ejs
    Mar 27, 2011 at 3:44
  • $\begingroup$ @Ejs - I'm afraid there's no short answer to your last question. You're getting into what it means to assess relationships when using statistical control - a fabulously intricate topic worthy of extensive study. You're also probably getting into the topic of variable selection, which is a big one as well. Imho the best source for the committed student of these topics is Pedhazur's amazon.com/Multiple-regression-behavioral-research-Pedhazur/… $\endgroup$
    – rolando2
    Mar 27, 2011 at 20:08
1
$\begingroup$

Let us say the interest lies in comparing two groups of people: those with $X_{1} = 1$ and those with $X_{1} = 0$.

The exponential of $\beta_{1}$, the corresponding coefficient, is interpreted as the ratio of the odds of success for those with $X_{1} = 1$ over the odds of success for those with $X_{1} = 0$, conditional on the other variables in the model.

So, if you have two models with different dependend variables then the interpretation of $\beta_{1}$ changes since it is not conditioned upon the same set of variables. As a consequence, the comparison is not direct...

$\endgroup$
2
  • $\begingroup$ Does this have any implications for roland2's suggestion? $\endgroup$
    – Ejs
    Mar 27, 2011 at 4:12
  • $\begingroup$ @Ejs. Do you refer to the standardisation step? By the way, does my answer help ? Have I misunderstood the question ? $\endgroup$
    – ocram
    Mar 27, 2011 at 6:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.