I am running regressions across a country's counties (N about 300). I divide the country in two regions A and B to control for potential unobservables. My explanatory variable varies at the county level and is relatively high in the counties of region A compared to region B (correlation between the variable and the regional dummy is about .7). My outcomes are observed either on the county, or at the within-county level (individuals living in a respective county). For some outcomes, my exp. variable is significant and the regional dummy is not, while for others, it is the other way round, which is fine. But sometimes, both of them are significant and the two point estimates are almost of the same magnitude, but with opposing signs, so they cancel each other out. I am not sure what to make of this - does it sound like two actual effects at work, or is it more of a sign that the regression is not able to differentiate between the effect of my exp. variable and the regional dummy?

Thanks in advance!

  • $\begingroup$ Is your EV continuous, binary, categorical? Have you tried interacting them to see if the effect of the EV depends on the region? Also are you using spatial weights? Have you estimated Moran's I to assess spatial heterogeneity? $\endgroup$ – robin.datadrivers Jan 23 '15 at 16:24
  • $\begingroup$ I have experimented with the EV - I have the data in continuous form, but when I categorized it, the significant effect always kicked in above X%, so I recoded it as a 0/1 dummy for treatment status. The correlation between the continuous or the binary form and the regional dummy is equally high. $\endgroup$ – HectorColossus Jan 23 '15 at 18:44
  • $\begingroup$ I've tried an interaction, but this interaction then has a corrrelation coefficient of around .9 with the EV and the regional dummy, so I guess this is not the way to go. The EV is aggregated from a lower level; in the process I've used spatial weights (share of a certain type of ground in total area) First time I here about Moran's I, thanks. But what would a rejection of the null imply for my empirical strategy? $\endgroup$ – HectorColossus Jan 23 '15 at 18:52
  • $\begingroup$ Moran's I is a test for autocorrelation, so if it is significant, you have spatial autocorrelation and should model with spatial weights (spatial regression models are really complicated in my view, so that is another challenge). An interaction term should be correlated with the EV, that is to be expected, since the interaction is made up of the EV. I wouldn't use that strictly to mean it's not important. Do some testing on it - is it statistically significant? Compare the AIC/BIC between the model with and without, do an LR test with the model with and without it, look at marginal effects. $\endgroup$ – robin.datadrivers Jan 23 '15 at 19:38
  • $\begingroup$ Can you provide more details on what you are specifically modeling? What is the EV? The DV? Where is your cut point to dichotomize the EV? $\endgroup$ – robin.datadrivers Jan 23 '15 at 19:39

To avoid a long discussion in the comments, here are some thoughts for you that can help uncover what is going on.

Countervailing effects of explanatory variables: This is perfectly normal when the variables are not measuring the same latent concept, even if they are highly correlated. For example, I am currently modeling grant placement by county (similar exercise as what you are doing). When I include median income, income has a positive relationship with grant placement. When I add population to the model (which has a moderate correlation with income), the effect of income becomes negative. Why? Because now I'm controlling for population. Before a 10% increase in income didn't hold anything else constant, so when income rises, so does population, and income was picking up both the effect of income and population. With pop in the model, an increase in income no longer picks up the population effect. You situation is likely similar. Without knowing more about the variables, I can't help you figure out if it makes sense, but this is the type of conceptual thinking you need to do to figure out if the modeling makes sense or not.

Interaction terms: I recommend you do some more digging on how interaction terms work. Interaction terms are going to be correlated with both components; and it is not uncommon because of that for an interaction to absorb the variance of the component parts. Don't think about interaction terms like you do other variables (where you worry about collinearity). The point of an interaction term is to identify if the effect of one variable is moderated by another variable. In your case, a policy intervention is likely to affect outcomes differently depending on other characteristics. Not just region in the country, but the specific characteristics of the counties. If the interaction term is significant, and you can interpret it, and the overall model fit improves, it is new information for you to consider. If you are using nonlinear models like logit, Poisson, etc., interactions can be tricky but easier to interpret with marginal effects.

Endogeneity: Though you don't provide a lot of detail on the policy intervention, I think you need to assume that the intervention may be a result, not a cause, of the dependent variables. That means a standard regression (be it linear, logit, etc.) will be biased. There are many different types of models you can look into that will control for this endogeneity - they have different assumptions and data requirements. Here are some examples:

  • Propensity score matching
  • Instrumental variables (2SLS)
  • Control functions (similar to instrumental variables)
  • Selection models (such as Heckman's selection model)
  • Time series/panel models, where you can clearly control for the temporal independence of the intervention and the effects

You will have a hard time convincing researchers of your results if you don't consider endogeneity.

  • $\begingroup$ I will start with: Endogeneity: My policy intervention actually took place some decades ago (by force). I am very confident to say that its allocation was not based on any socio-economic factors, but simply on the question whether a county was in region A or B. So I feel safe about endogeneity as long as I can clearly separate the effect of the policy from let's say potential traits of the population in A which might affect the outcomes, which is what I'm struggling with. $\endgroup$ – HectorColossus Jan 24 '15 at 13:33
  • $\begingroup$ Interactions: Good remarks. But I am still unsure whether a significant interaction implies that either the policy only works together with particular characteristics of region A, or the significance arises from just the intensity of the policy increasing so much when 'moving' into region A. Countervailing effects: Helpful remarks as well. My scepticism is based in particular on the result that the policy effect and the regional effect quite often exactly cancel out and I cannot figure out whether to attach a deeper meaning to that, or to just see it as a zero finding. $\endgroup$ – HectorColossus Jan 24 '15 at 13:37
  • $\begingroup$ I think unfortunately without seeing the results and understanding the data and context more, I can't really provide further comments. If you want to go more in depth, feel free to email me (found in my profile). Otherwise, good luck with this, sounds like an interesting project. $\endgroup$ – robin.datadrivers Jan 24 '15 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.