On page 9 of "High Dimensional Sparse Econometric Models: An Introduction (2011)," Belloni and Chernozhukov explain in Remark 1 that the expected risk of a sparse estimator is $$\min_{\beta\in\mathbb{R}^{\tilde{T}}} \mathbb{E}_{n}[(f_{i}-x_{i}[\tilde{T}]'\beta)^{2}]+\sigma^{2}\dfrac{k}{n}.$$ The notation is explained in the paper, and is standard. $x_{i}[\tilde{T}]$ denotes a vector of values (say, $k$) of some subset of predictors, generally smaller than $p$, the number of all potential predictors. I fail to understand why. Help would be appreciated. Here is direct arXiv link to the paper.

  • $\begingroup$ Since $\beta$ is a vector of length p ($\beta \in \mathbb{R}^p$) then necessarily $x_i[\tilde T]$ is a vector of length p as well, or their inner product could not be formed. Could you clarify your question? $\endgroup$ – jwimberley Jan 14 '17 at 21:41
  • $\begingroup$ You are right, that was a typo on my part. Corrected. $\endgroup$ – rsm Jan 14 '17 at 21:51

Let $\Pi$ a projection matrix. Then, \begin{align*} \frac{1}{n} \, \mathbb{E} \| \Pi y - f^* \|_2^2 & = \frac{1}{n} \, \mathbb{E} \|\Pi f^* - f^* - \Pi \epsilon \|_2^2 \\ & = \frac{1}{n} \, \mathbb{E} \|\Pi f^* - f^*\|_2^2 + \frac{1}{n} \, \mathbb{E} \|\Pi \epsilon \|_2^2 \\ & = \frac{1}{n} \, \mathbb{E} \|\Pi f^* - f^*\|_2^2 + \frac{1}{n} \, \mathbb{E} \, \textrm{trace} \left( \|\Pi \epsilon \|_2^2 \right) \\ & = \frac{1}{n} \, \mathbb{E} \|\Pi f^* - f^*\|_2^2 + \frac{1}{n} \, \mathbb{E} \, \textrm{trace} \left( \epsilon^T \Pi \epsilon \right) \\ & = \frac{1}{n} \, \mathbb{E} \|\Pi f^* - f^*\|_2^2 + \frac{1}{n} \, \mathbb{E} \, \textrm{trace} \left( \epsilon \epsilon^T \Pi \right) \\ & = \frac{1}{n} \, \mathbb{E} \|\Pi f^* - f^*\|_2^2 + \frac{\sigma^2}{n} \, \textrm{trace} \left( \Pi \right) \\ & = \frac{1}{n} \, \mathbb{E} \|\Pi f^* - f^*\|_2^2 + \frac{\sigma^2}{n} k, \\ \end{align*} where $k$ is the dimension of the space onto which $\Pi$ is projecting.

This is the "bias$^2$ + variance" decomposition of the risk, and it appears to be the computation behind their equation.

I should note that this computation only reveals that this decomposition holds when the estimate $x^T \beta$ is a projected form of the signal $f^*$---however, this won't necessarily be the case, since they're using the form $\Pi f^* = x^T \beta$ for an arbitrary $\beta$ (which they then minimize over.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.