4
$\begingroup$

I have this problem.

I have Y (market share) and X (store size).

I want to predict Y from X using a linear regression ... I run OLS to find the betas, their pvalue is meaningful, yada, yada, yada ... But can I use OLS in the first place?

I took an econometrics course in college, and I remember the lecturer saying that when the explanatory variable can be explained by the variable you are trying to explain (in this case, store size may be in some cases the result of market share), then you cannot use OLS as an estimator. Or you can, but only after a two-step estimation where you use something else as an instrumental variable.

I never understood why this was so, and I continue to struggle to find examples of this happening, and most statisticians I deal with do not seem to care about this econometric stuff ... Can anyone confirm for me that causation problems such as this means the OLS estimator might be inconsistent? If so, do you guys have an example where this happens and what it means?

$\endgroup$
  • $\begingroup$ in what sense do you mean that the p-value is 'meaningful'? $\endgroup$ – Glen_b -Reinstate Monica Mar 25 '15 at 23:50
  • $\begingroup$ pvalue < 0.05. The statistics software I know present the coefficients of the model along with a pvalue, which I believe is calculated using the F-test. $\endgroup$ – Ricardo Cruz Mar 26 '15 at 14:31
3
$\begingroup$

You want to estimate the true causal effect of $X$ on $Y$. You are asking this question:

But you might remember, when first learning about causality and causal confounders, that an unmeasured prior cause of $X$ and $Y$, say, $U$, will confound $E(Y|X)$ as an unbiased estimator of the true causal effect of $X$ on $Y$.

If "the explanatory variable can be explained by the variable you are trying to explain" then you have a situation where past values of $Y$ cause current values of $X$, which then may in turn cause future value of $Y$. If we make time explicit in our causal graphs, you can see how past values of $Y$ confound probability of future values of $Y$ conditional on current values of $X$ (assuming that past values of $Y$ cause future values of $Y$ via some pathway not involving $X$):

So can you just condition on $Y_{t=1}$ to eliminate this confounding? No, because there will be values of $X$ and $Y$ farther in the past that will bias the conditioning on $Y_{t=1}$ (for example). This is why instrumental variables are so cool (when you can find them):

Notice that $Z$ is not caused by past values of $X$ or $Y$ (the dotted arrow suggests that $Y$ is so confounded by past values), so $E(Y_{t=3}|\hat{X}_{t=2})$, where $\hat{X}_{t=2} = E(X_{t=2}|Z_{t=1})$ is an unbiased estimator of the true causal effect of $X_{t=2}$ on $Y_{t=3}$ (if the causal assumption that $Z_{t=1}$ causes $Y_{t=3}$ only through $X_{t=2}$ holds).

Elaborations on the model could, naturally generalize to $X$ and $Y$ at many times.

$\endgroup$
  • $\begingroup$ Very cool way to put it, in terms of a time series. From the wikipedia, "Bias is related to consistency as follows: a sequence of estimators is consistent if and only if it converges to a value and the bias converges to zero." So, the problem is that, as I increase my sample, the parameters estimated in my model may vary widely, is that so? Is that why having these problems in mind when choosing the estimator is a good idea? (Also, just to finish off hehe, what hypothesis test would be good to prevent this? (I mean, to know whether there could be causality issues.)) $\endgroup$ – Ricardo Cruz Mar 25 '15 at 19:45
  • 1
    $\begingroup$ @RicardoCruz I don't think of it in terms of "vary widely" but in terms of bias (sample size does not correct bias from causal confounding). As to what model (not test) will correct for this bias, you said it yourself: instrumental variables models! Also: if you like my answer please feel free to vote on it by clicking the up-arrow. You ought to also click the check mark next to the answer that you feel best answers your question (if any). Since I am not in economics I can;t suggest good instrument(s) for your variables. $\endgroup$ – Alexis Mar 25 '15 at 19:49
  • $\begingroup$ Thank you, Alexis. So, I think my confusion was that the term inconsistency to me means that my parameter estimation may vary according to sample size (ie. if I run OLS with a given sample, I get a certain beta vector, but if I run it with another sample may get a very different beta vector). I thought it was inconsistent because it did not converge. You are saying it is inconsistent because it is unbiased. The two conditions are necessary for consistency, and my confusion was that I thought the first was the problem here. $\endgroup$ – Ricardo Cruz Mar 25 '15 at 19:55
  • $\begingroup$ @RicardoCruz biased not unbiased. :) $\endgroup$ – Alexis Mar 25 '15 at 19:58
  • $\begingroup$ Just one thing: what does it mean in practice the model being biased, as long as it seems to have good prediction value in the validation tests? $\endgroup$ – Ricardo Cruz Mar 25 '15 at 20:04
4
$\begingroup$

This is called simultaneity bias. One example is the effect of police on crime. Cops reduce crime (hopefully), but mayors also hire cops when crime wave hits. You have both criminals and mayors making their decisions simultaneously, but you only observe the outcome of the joint decision.

$\endgroup$
  • $\begingroup$ Dimitriy, I understand the examples. What I do not understand is what does it mean in practice... So, if I have number-of-crimes~number-of-cops, and I stupidly run an OLS on that, we were told that the result was "inconsistent" or something along those lines... What does it mean in practice? That, if I run the test in another city I could get different results? Is that what inconsistency means? $\endgroup$ – Ricardo Cruz Mar 25 '15 at 18:56
  • $\begingroup$ What I find strange is that we were told the estimation would be inconsistent even when using the same sample... I never quite got over that... What does this means in practice? If I use a subset of the sample, or if I enlarged the sample with more data, the betas estimation could be way different? Is that what it means? (And why do my other colleagues from mathematics who never attended this econometrics course in the econ dept, do not seem concerned about any of this?) $\endgroup$ – Ricardo Cruz Mar 25 '15 at 19:01
  • 2
    $\begingroup$ It means that if even if you get lots of data and you know the basic form of the crime supply equation (say, crime=a+b*cops), you will not estimate the true behavioral parameter b that you seek. You will certainly get some number from OLS, but when you use it to say what happens to crime when you fire/hire more cops, your predictions will be quite poor. $\endgroup$ – Dimitriy V. Masterov Mar 25 '15 at 21:21
  • 1
    $\begingroup$ I'm upvoting your answer to draw attention to your comment, which I think most directly deals with what I think the OP hasn't understood - that the obtained estimate doesn't actually tell him what he hopes it does. $\endgroup$ – Glen_b -Reinstate Monica Mar 25 '15 at 23:48
  • 1
    $\begingroup$ @Glen_b, yes. The term "consistency" was confusing me, because I was interpreting it too literally. I took it as meaning that the estimations could vary widely. This entire discussion makes things much more clear. The problem is one of bias (and, by definition, consistency, because consistency is defined as converging to the real parameter, which it doesn't, because it does potentially converge but to a biased parameter). $\endgroup$ – Ricardo Cruz Mar 26 '15 at 14:35
3
$\begingroup$

You will find one of my earlier answers useful.

Imagine I had a dataset containing observations on people who suspected that they had the flu. In the data are two variables, $T$, their body temperature, and $V$, the viral load of influenza (measured with a lab test). You run the following regression: \begin{align} V = \beta_1 + \beta_2T + \epsilon \end{align} You find a large, positive value for $\hat{\beta}_{2,OLS}$. You conclude that a good way to treat the flu is to put people in ice baths to reduce their body temperature. Right?

Not right. In fact, idiotic. Both the person's viral load test result and the person's body temperature are being affected by whether and how badly they are infected with influenza. In fact, the following regression is probably closer to right: \begin{align} T = \alpha_1 + \alpha_2V + \nu \end{align} Your body temperature is affected by how much virus you have.

As I say in the earlier answer linked above, we run regressions for two totally different purposes. First, we run them for prediction. If you wanted to predict viral load based on body temperature, the first regression above is a fine way to do it. If you wanted to predict temperature based on viral load, the second one would be a fine way to do it.

Second, we run them to understand causation. If my question is "How much would viral load fall if I reduced a patient's body temperature by 1 degree (in an ice bath)," I can run my first regression above. But the answer it gives me is only right (consistent) if the equation captures causation properly. If the causation, out in the actual world, runs from body temperature to viral load (and not the other way, and not from some third factor to both temperature and viral load).

If my question is "How much would temperaure fall if I reduced a patient's viral load by 1 unit (say via an anti-viral drug)," I can run my second regression. But the result it gives me is only right (consistent) if the equation captures causation properly. If the causation, out in the actual world, runs from viral load to body temperature (not the other way, and not from some third factor to both temperature and viral load).

In your example, I assume the business question you are interested in is something like "What will happen to a store's market share if I increase its size by 1000 square feet?" If you try to answer this question with the regression you propose, you will, I am almost certain, get an answer which is much too big.

What do I mean, too big? Suppose you got a coefficient of 10 (measuring size in thousands of square feet and market share in percents). That would appear to say that increasing store size by 1000 square feet would increase market share by 10 percentage points. If you then run an experiment, out in the real world, in which you increase half (chosen randomly) of the stores' sizes by 1000 square feet, the increase in their market shares (compared to the not-increased stores) will be less than 10 percentage points.

How do I know this? Because I am confident that there is a third factor which affects both store size and market share. That third factor is the demand for that store. Somebody decided to make store 1 7400 square feet and store 2 4400 square feet. He did that because he thought store 1 would have a higher demand than would store 2. Likely, store 1 also has a higher market share than does store 2. Again, because it has higher demand. So, size and market share will be positively correlated. But not only or even primarily because size causes market share. More because market share causes size.

Wait, you say! If I can use my regression for prediction, then I'll use it to predict market share for a 6000 square foot store and then again for a 7000 square foot store. The difference in the two predictions tells me how much market share would rise if I make the 6000 square foot store 1000 square feet bigger!

No, it does not work that way. The prediction you make using estimates from dataset is only good for circumstances like the ones in the data. If the variation in the data in store size came from Ricardo Cruz doing experiments, then the method described in the previous paragraph will work fine. On the other hand, if the variation in the data in store size come from some guy making guesses about "how big a store do we need here," then these predictions are not valid in a new, different dataset where the variation in store size comes from Ricardo Cruz doing experiments.

$\endgroup$
  • $\begingroup$ I didn't expect so many people answering, so I already gave the points to someone else, but thank you very much for going all the way to answer my newbie confusion. By the way, do you recommend any hypothesis test for testing causality issues? I seem to recall "Wald" test ... $\endgroup$ – Ricardo Cruz Mar 25 '15 at 23:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.