4
$\begingroup$

Francis Diebold has a blog post "Causality and T-Consistency vs. Correlation and P-Consistency" where he presents the notion of P-consistency, or presistency:

Consider a standard linear regression setting with $K$ regressors and sample size $N$. We will say that an estimator $\hat\beta$ is consistent for a treatment effect ("T-consistent") if $$ \text{plim} \ \hat\beta^k = \frac{\partial E(y|x)}{\partial x_k}, $$ $\forall k=1,\dots,K$; that is, if $$ \left( \hat\beta_k−\frac{\partial E(y|x)}{\partial x_k} \right) \xrightarrow{p} 0, $$ $\forall k=1,\dots,K$. Hence in large samples $\hat\beta_k$ provides a good estimate of the effect on $y$ of a one-unit "treatment" performed on $x_k$. T-consistency is the standard econometric notion of consistency. Unfortunately, however, OLS is of course T-consistent only under highly-stringent assumptions. Assessing and establishing credibility of those assumptions in any given application is what makes significant parts of econometrics so tricky.


Now consider a different notion of consistency. Assuming quadratic loss, the predictive risk of a parameter configuration $\beta$ is $$ R(\beta)=E(y−x′\beta)^2. $$ Let $B$ be a set of $\beta$'s and let $\beta^∗\in B$ minimize $R(\beta)$. We will say that $\hat\beta$ is consistent for a predictive effect ("P-consistent") if $$ \text{plim} \ R(\hat\beta)=R(\beta^∗); $$ that is, if $$ (R(\hat\beta)−R(\beta^∗)) \xrightarrow{p} 0. $$ Hence in large samples $\hat\beta$ provides a good way to predict $y$ for any hypothetical $x$: simply use $x′\hat\beta$. Crucially, OLS is essentially always P-consistent; we require almost no assumptions.

<...>

The bottom line: In sharp contrast to T-consistency, P-consistency comes almost for free, yet it's the invaluable foundation on which all of (non-causal) predictive modeling builds. Would that such wonderful low-hanging fruit were more widely available!

Questions:

  1. What are the conditions under which P-consistency holds?
  2. Simple counterexample(s) where P-consistency does not hold
  3. Does presence of T-consistency imply presence of P-consistency?
$\endgroup$
  • $\begingroup$ Note to self: Hayashi "Econometrics" section 2.9 "Least squares projection" could be relevant. Specifically: That is, under Assumption 2.2 (ergodic stationarity) and Assumption 2.4 guaranteeing the nonsingularity of $\mathbb{E}(XX^\top)$, the OLS estimator is always consistent for the projection coefficient vector, the $\beta^*$ that satisfies the orthogonality condition (2.9.5). $\endgroup$ – Richard Hardy Feb 9 at 17:57
2
$\begingroup$

This post by Francis is conflating causal concepts with statistical (regression) concepts.

First, the definition of a treatment effect is incorrect: a causal effect is defined in terms of a causal model, not in terms of the observed joint probability distribution.

The quantity $\frac{\partial E[y|x]}{\partial x_k}$ is not a treatment effect, it is the marginal change in the expected value of $y$ when we observe a change in $x_{k}$. For any linear conditional expectation function, this quantity is always the population regression coefficient of $x_k$.

A treatment effect estimand would make reference to interventions or counterfactuals,such as $\frac{\partial E[y_{x}]}{\partial x_k}$ in counterfactual notation, or $\frac{\partial E[y|do(x)]}{\partial x_k}$ in $do()$ notation.

Thus, his statement, “the distinction between P-consistency and T-consistency is clearly linked to the distinction between correlation and causality” is total nonsense.

Now to more precisely answer your questions:

1) By definition, OLS is the solution to minimizing $R(\beta)$, so it always holds under the usual standard assumptions that you have i.i.d samples from a stationary (well behaved) process.

2) Your RW example is not really valid, because a random walk is not stationary. I’m saying this because otherwise a counterexample could be just: “Imagine you have n observations, each coming from different arbitrary distributions.” In the RW case, you could estimate a specific time point regression coefficient consistently with OLS if you had infinite samples from that time point. Thus, to make this question meaningful, you would need to properly define the constraints, such as in Wasserman’s post, where he confined the coefficients to a l1 ball, and let the dimension of the ball grow with the sample size. And note this has nothing to do with causality.

3) As explained, in the usual set-up P-consistency always holds regardless of whether $E[y|x]$ is linear, and thus whether the OLS coefficients corresponds to $\frac{\partial E[y|x]}{\partial x_k}$ (which is not the definition a "treatment effect", as Francis incorrectly states).

$\endgroup$
  • $\begingroup$ Thank you for the very relevant an interesting points! These are very helpful for my understanding. At the same time, your answers to the specific questions are not quite sufficient / not concrete enough. In (1), is i.i.d samples from a stationary process the condition? Could you be more specific? Usually, I encounter either i.i.d. or stationary, not both (though i.i.d. implies stationarity). In (2), the example seems to be valid precisely because it is a counterexample which I am asking for. It specifically violates the stationarity assumption. Do you think this does not make sense? $\endgroup$ – Richard Hardy Nov 6 '18 at 6:11
  • $\begingroup$ In (3), could you elaborate a bit more? Could you specify the conditions under which T-consistency implies P-consistency? Thanks again! $\endgroup$ – Richard Hardy Nov 6 '18 at 6:12
  • $\begingroup$ @RichardHardy thanks for pinging, I had forgotten about it! I have a deadline coming up, but after will come back to complement the answer. $\endgroup$ – Carlos Cinelli Jan 16 at 7:31
  • $\begingroup$ @RichardHardy back here! So, regarding (i) my main point is that by definition the population regression coefficients are always identifiable. You do not need i.i.d necessarily, the most well known case is when you have heteroskedastic data, or some forms of autocorrelation, in which the data is not iid, yet consistency still holds. I believe there is a paper spelling out necessary conditions for consistency under a general set-up, I will try to find it. But my main point is that since OLS is the minimizer of $R(\beta)$ in the population, consistency generally holds barring pathological cases. $\endgroup$ – Carlos Cinelli Feb 4 at 23:40
  • $\begingroup$ @RichardHardy Regarding (iii) if the conditional expectation is linear, the population regression coefficients are also the derivatives of the conditional expectation. Thus, in this case consistency of $\beta$ implies consistency of the derivatives of the conditional expectation, the only extra assumption is linearity of $E[Y|X]$. In the non-linear case, Francis question is underspecified, since to start we must define better what the target is (the derivative at one point X = x^*, the derivative averaged over the other X?). $\endgroup$ – Carlos Cinelli Feb 4 at 23:55
1
$\begingroup$
  1. --- ? ---
  2. P-consistency will not hold when regressing two independent random walks on each other. In this setup, the OLS estimator converges to a random variable rather than the true parameter value (which is zero).
    (Then w.r.t. 1., we need some sort of stationarity or moment conditions for P-consistency.)
  3. --- ? ---
$\endgroup$
  • 1
    $\begingroup$ Besides you I don't think many people on this site have the background or knowledge required to answer such an advanced question ! Is there some form of "canonical" or base question regarding risk consistency ? Here are a set of notes which are very good from Tibshirani, in particular sparcity and linear regression stat.cmu.edu/~ryantibs/statml/lectures $\endgroup$ – Xavier Bourret Sicotte Sep 13 '18 at 6:12
  • $\begingroup$ @XavierBourretSicotte, I will take this as a compliment, thank you. However, I know quite a few people on this site who are clearly more knowledgeable and/or experienced than I am. I am sure more than one of them could answer the question if they tried hard enough. The problem might be that either they are not interested in the question or they do not have the time to try hard enough (or simply they have not seen the question – the view count is quite low). Regarding a canonical question regarding risk consistency, I do not know the answer, sorry. $\endgroup$ – Richard Hardy Sep 13 '18 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.