11
$\begingroup$

I've developed a logit model to be applied to six different sets of cross-sectional data. What I'm trying to uncover is whether there are changes in the substantive effect of a given independent variable (IV) on the dependent variable (DV) controlling for other explanations at different times and across time.

My questions are:

  • How do I assess increased / decreased size in the association between the IV and DV?
  • Can I simply look at the different magnitudes (sizes) of the coefficients across the models or do I need to go through some other process?
  • If I need to do something else, what is it and can it be done/how do I do it in SPSS?

    Also, within a single model,

  • Can I compare the relative size of independent variables based on unstandardised scores if all are coded 0-1 or do I need to convert them to standardised scores?
  • Are there problems involved with standardised scores?
$\endgroup$
  • 2
    $\begingroup$ This article may be of interest, dx.doi.org/10.1093/esr/jcp006 , apparently comparing effects between Logistic models is considerably more complicated than in the OLS case! $\endgroup$ – Andy W Mar 25 '11 at 17:17
13
$\begingroup$

I will mainly focus on your first three questions. The short answers are: (1) you need to compare the effect of the IV on the DV for each time period but (2) only comparing the magnitudes can lead to wrong conclusions, and (3) there are many ways of doing that but no consensus on which one is correct.

Below I describe why you cannot simply compare coefficient magnitudes and point you to some solutions that have been thought of so far.

According to Allison (1999), unlike OLS, logistic regression coefficients are affected by unobserved heterogeneity even when such heterogeneity is not related to the variable of interest.

When you fit a logistic regression like:

(1)$$ \ln\bigg(\frac{1}{1-p_i}\bigg) = \beta_{0} + \beta_{1}x_{1i} $$

You are in fact fitting an equation predicting the value of a latent variable $y^*$ that represents the underlying propensity of each observation to assume the value $1$ in the binary dependent variable, what happens if $y^*$ is above a certain threshold. The equation for that is (Williams, 2009):

(2)$$ y^* =\alpha_{0} + \alpha_{1}x_{1i} + \sigma \varepsilon $$

The term $\varepsilon$ is assumed to be independent from the other terms and to follow a logistic distribution – or a normal distribution in the case of probit and a log-logistic distribution in case of complementary log-log and a cauchy distribution in the case of cauchit.

According to Williams (2009), the $\alpha$ coefficients in equation 2 are related to the $\beta$ coefficients in equation 1 through:

(3)$$ \beta_{j} = \frac{\alpha_{j}}{\sigma}\;\;j=1,...,J. $$

In equations 2 and 3, $\sigma$ is the scaling factor of the unobserved variation, and we can see that the size of the estimated $\beta$ coefficients depends on $\sigma$, which is not observed. Based on that, Allison (1999), Williams (2009), and Mood (2009), among others, claim that you cannot naively compare coefficients between logistic models estimated for different groups, countries or periods.

This is because comparisons may yield incorrect conclusions if the unobserved variation differs between groups, countries or periods. Both comparisons using different models and using interaction terms within the same model suffer from this problem. Besides logit, this also applies to its cousins probit, clog-log, cauchit and, by extension, to discrete time hazard models estimated using these link functions. Ordered logit models are also affected by it.

Williams (2009) argues that the solution is to model the unobserved variation through a heterogeneous choice model (a.k.a., a location-scale model), and provides a Stata add on called oglm for that (Williams 2010). In R, heterogeneous choice models can be fit with the hetglm() function of the glmx package, which is available through CRAN. Both programs are very easy to use. Lastly, Williams (2009) mentions SPSS's PLUM routine for fitting these models, but I have never used it and cannot comment in how easy it is to use.

However, there is at least one working paper out there showing that comparisons using heterogeneous choice models can be even more biased if the variance equation is misspecified or there is measurement error.

Mood (2010) lists other solutions that do not involve modelling the variance, but use comparisons of predicted probability changes.

Apparently it is an issue that is not settled and I often see papers in conferences of my field (Sociology) coming up with different solutions for it. I would advise you to look at what people in your field do and then decide how to deal with it.

References

$\endgroup$
  • $\begingroup$ I am trying to implement Williams (2009) solution in R and it seems that the new versions of the glmx package does not have the hetprob() function anymore. Just wanted to check if you know any alternatives for this? $\endgroup$ – AliCivil Jan 11 '16 at 0:09
  • 1
    $\begingroup$ I haven't used glmx in a while and didn't know that it had changed. Now it is available through CRAN, and the function used for hereroskedastic probit is called hetglm( ), apparently. I'll update this answer to reflect this later on (it is about bed time here). I hope that this helps for now. $\endgroup$ – Kenji Jan 11 '16 at 0:41
3
$\begingroup$

Are there changes across data sets? I can answer that without seeing the data! Yes. There are. How big are they? That's key. For me, the way to see is by looking. You will have odds ratios for each independent variable for each data set - are they different in ways people would find interesting? Now, it's true each will have a standard error and so on, and there are probably ways to see if they are statistically significantly different from each other, but is that really an interesting question? If it is, then one way to test it easily with software would be to combine all the studies, and include "study" as another independent variable. You could then even test interactions, if you wanted. Whether you want to do this depends on your substantive questions.

As to comparing variables within a model, the main problem with standardized scores is that they are standardized on your particular sample. So, the parameter estimates and so on are then in terms of standard deviations of the variables in your particular sample. Even if your sample is truly a random sample from some population, it will have (slightly) different standard deviations from other random samples. This makes things confusing.

The other problem is what the question of "relative size" even means. If your IVs are things that are well-understood, you can compare the ORs across ranges that mean something.

$\endgroup$
  • $\begingroup$ That useful, thanks Peter. The reason I asked the first question is because I've seen exactly that - comparisons of coefficients across models within a sample and comparisons across models from different samples - in refereed papers. I didn't feel it was the right approach, and, evidently, I'm correct. Regarding the technical details, can't I simply estimate a model on all six samples and an interaction term between the key predictor that I want to compare and a variable specifying each sample (representing a different time period)? Is that what you were saying? Do I need a variable for each s $\endgroup$ – Ejs Mar 25 '11 at 9:14
  • $\begingroup$ Hi @ejs. You would need to code "sample" the same way as any other categorical variable - dummy coding or effect coding or whatever. $\endgroup$ – Peter Flom Mar 25 '11 at 9:52
  • $\begingroup$ Regarding interactions .... yes, they can be difficult to interpret. I like a graphical approach to showing what they mean. $\endgroup$ – Peter Flom Mar 25 '11 at 9:53
3
$\begingroup$

Guilherme is on the money here. While the other responses are useful, please note that logistic regression (and all nonlinear regression like Poisson, for that matter) are fundamentally different than linear regression. There may be serious issues with the logit scaling factor when running the same analysis on six different data sets and then running that analysis on the combined data set. Changes in coefficients may have nothing to do with meaningful differences (even if statistically significant or substantively important). They could have everything to do with unobserved heterogeneity across the samples. You absolutely have to test for that. Many (if not most) researchers in social and policy science fields ignore this. Guilherme gives the seminal articles on this that I recommend everyone look at. Peters suggestions are practical, but simply coding a dummy variable for the sample the data come from will not address this heterogeneity in the scaling factor. You can do that in linear regression and the heterogeneity shouldn't affect your coefficients, but here it may.

One other aspect to the effect of unobserved heterogeneity unique to logit vs. linear regression, is the effect of different regressors in each data set. If you don't have the same variables, or likely if they are measured differently, you have a form of omitted variable bias. Unlike with linear regression, an omitted variable orthogonal to your key regressor can still bias your estimate. As Cramer puts it:

Even with orthogonal regressors, then, omitted variables depress $\hat β$ towards zero, relatively to its value in the full equation. In other words, the $\hat β$ of discrete models vary inversely with the extent of unobserved heterogeneity. The practical consequence is that estimates from samples that differ in this respect are not directly comparable. (http://dare.uva.nl/document/2/96199)

Cramer also points out though the coefficient estimates are biased downward when omitting a variable, the partial derivatives are not. This is fairly complicated and you should read the article for a more lucid explanation – the overall point is, don't exclusively look at the log-odds or odds ratios. Consider predicted probabilities and derivatives; see the margins command in Stata for more details. JD Long has a paper that goes into detail here.

Finally, there are a number of papers you can Google for that discuss interaction terms in logit models. My understanding has been that take the logit coefficient on an interaction as a guide but not definitive, especially if you prefer to see the coefficients as exponentiated odds ratios. Looking at predicted probabilities and average marginal effect is better (again, look up documentation on Stata's margin command for logit, even if you use SPSS this will still be helpful).

I'm not familiar enough with SPSS to know how that package can deal with these issues, but I will say this: when you get into deeper statistical issues like this, it is an indication that it is time for you to move to a more flexible, sophisticated package like Stata or R.

$\endgroup$
  • $\begingroup$ +1 for recommending marginal effects and for recommending moving into R. $\endgroup$ – Kenji Jan 21 '15 at 14:58
1
$\begingroup$

Another tool that may be useful is the standardarized regression coefficient, or at least a rough-and-ready pseudo-version. You can obtain one such version by multiplying your obtained coefficient by the standard deviation of the predictor. (There are other versions and some debate about the best one, e.g. see Menard 2002, Applied Logistic Regression Analysis (Google books)). This will give you a way to assess the strength of the effect across studies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.