8
$\begingroup$

I just got a rejection from an economics journal. Among the reasons cited for rejection were:

the benefits of using the semi-parametric method are not clearly brought out compared to alternative simpler techniques with clean identification of causal relationships

It is certainly possible that I could have done a better job of motivating the methodology to a bunch of economists who generally stick to OLS. But have I violated "clean identification"? Please judge for yourself and let me know what you think:

My main estimating equation is $$ y_{it} = \alpha_i + \beta_1 T_{it} + f\left(\begin{array}{l}Z_{it}\\ Z_{it} \times T_{it} \\ Z_{it}\times T_{it} \times X_t\end{array} \right) + \beta_2X_t + \epsilon_{it} $$ $Z$ is continuous, $X$ and $T$ are binary. I can justifiably assume that $$ E[\epsilon|\alpha,T] = 0 $$ Which is to say that the coefficient on $T$ is unbiased conditional on individual-level dummy variables ("fixed effects" in econometrics-speak). When I include continuous variable $Z$, I am simply looking at heterogeneity in estimated treatment effects over gradients of $Z$. So the average causal effect of the treatment $T$ is a an average of $\hat\beta_1 + \hat f_{Z\times T}$ for the various levels of $Z$ that I observe.

The model is etimated by penalized quadratic splines (e.g.: Ruppert et al. 2003). Specifically: $$ y = \beta_0 +X'\beta + \displaystyle\sum_{1}^p (Z^{p})'\gamma + \displaystyle\sum_{j=1}^{\#vars} \displaystyle\sum_{k=1}^{\# knots_j}\delta_{jk}\left(\left(Z_j - \kappa_{jk} \right)^p \times \left(Z_j > \kappa_{jk} \right)\right) + \epsilon $$

This is solved by $$ \left[\begin{array}{c} \hat\beta\\ \hat\gamma \\ \hat \delta \\ \end{array}\right] = (C'C + \lambda^{2p}D)^{-1}C'y $$

where $C$ includes the parametric terms and the knot terms, and where the ridge penalty $\lambda$ only applies to the knot terms, and is chosen to minimize AIC. (I can't do full justice to the methodology -- see Ruppert et al, or Simon Wood's textbook on GAM's).

Of course, I use these semiparametrics because I don't want to impose unfounded functional forms onto my data. Doing so would quite naturally bias my estimates as much as imposing a logarithmic fit onto a sinusoidal function would bias my estimates. But is there something inherent in penalized splines as I've described them that would inherently render the following statement untrue?

$$ E[\hat\beta_1] = \beta_1 \text{ iff } E[\epsilon|\alpha,T] = 0 $$

$\endgroup$
  • 2
    $\begingroup$ I'm not qualified to answer your final question (although it seems suspicious), but perhaps to address the Journals concerns you should also include a OLS model in your paper and show that it performs badly by some metric? $\endgroup$ – thebigdog Jun 4 '13 at 4:56
  • $\begingroup$ You did not violate "clean identification." There's nothing inherent that makes semi-parametric model less capable of achieving clean identification. Indeed, your model encompasses a linear model. $\endgroup$ – user28544 Jul 28 '13 at 5:22
  • $\begingroup$ @generic_user did you ever receive a resolution to this? If so, can you answer your question? If not, could you provide a definition of clean identification? I have some perspectives on publishing spline-adjusted analyses that may or may not be pertinent to this case. $\endgroup$ – AdamO Jul 20 '17 at 20:23
  • $\begingroup$ Late to the party, but I think you're worrying about the wrong thing here. The refs are saying they don't like that you added complexity without proving that it's useful. An example showing a failure mode of their go-to simple methods would help motivate the extra complexity you're introducing. It should be possible to engineer (or even better find a real world example) of where splines are needed to properly identify a causal relationship. $\endgroup$ – Paul Nov 13 '17 at 22:57
  • $\begingroup$ If this was published as some point can you please mention the name of the paper? It seems like an interesting application. $\endgroup$ – usεr11852 says Reinstate Monic May 6 '18 at 21:00
1
$\begingroup$

"Clean identification" of regression parameters is not an established concept. I believe what the reviewer means by this is that you should specify a parameter which is interpretable, testable, of low dimensionality, and for which the analysis is decently powered to detect so that an unbiased estimate can be obtained with relatively good efficiency.

The desire for "clean identification" does not imply OLS is the only suitable tool for the job. OLS is, however, a theoretically and practically sound tool for specifying and estimating parameters under a variety of settings. The desire for "clean identification" does not preclude semiparametric inference either. As a note, the spline extends an OLS model by creating (a) complex representation(s) of covariates. Semiparametric inference involves flexible modeling to eliminate the influence of ancillary statistics, but in your model it seems the main exposure is handled in such a fashion.

I think the reviewer raises two substantiated concerns. First is the rationale for penalization. Penalized regression methods are valuable for prediction. They are rarely used for inference. Penalized methods like ridge regression are biased, and it is difficult to describe or assess the bias. The goal of minimizing AIC is to obtain the best predictions, not valid inference. The second substantiated concern is whether the spline is even necessary to model the main exposure. It is true as you say that a spline is capable of modeling complex nonlinear functional forms. However, a spline simplifies very little. It is a complex high dimensional representation, with knot points and tuning that can be a source of researcher bias, and covariates that are nearly uninterpretable for anyone except highly trained statisticians. Many statistically significant trends that are precisely modeled by splines have underlying linear approximations which are neither statistically nor practically significant. Many statisticians and field experts agree that, in that case, both results should be carefully reported and/or that the spline is committing a type I error.

If the functional form of the main exposure is misspecified, it is possible to use Huber White standard errors to obtain consistent and unbiased inference for the least squares slope as a first order approximation to any non-linear trend. Splines can be used to model precision variables, on which you do not base inference, when there is a complex design to the data. This serves to effectively match and reduce variability when there is complex heterogeneity in data.

I think the reviewers comments can be addressed by fitting a linear model for the exposure and conducting inference with Huber White Sandwich errors. If the inference mostly agrees with the spline inference, comment on the spline model insofar as it demonstrates a curvilinear trend between the exposure and the response.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.