11
$\begingroup$

Suppose I have two estimators $\widehat{\beta}_1$ and $\widehat{\beta}_2$ that are consistent estimators of the same parameter $\beta_0$ and such that $$\sqrt{n}(\widehat{\beta}_1 -\beta_0) \stackrel{d}\rightarrow \mathcal{N}(0, V_1), \quad \sqrt{n}(\widehat{\beta}_2 -\beta_0) \stackrel{d}\rightarrow \mathcal{N}(0, V_2)$$ with $V_1 \leq V_2$ in the p.s.d. sense. Thus, asymptotically $\widehat{\beta}_1$ is more efficient than $\widehat{\beta}_2$. These two estimators are based on different loss functions.

Now I want to look for some shrinkage techniques to improve finite-sample properties of my estimators.

Suppose that I found a shrinkage technique that improves the estimator $\widehat{\beta}_2$ in a finite sample and gives me the value of MSE equal to $\widehat{\gamma}_2$. Does this imply that I can find a suitable shrinkage technique to apply to $\widehat{\beta}_1$ that will give me the MSE no greater than $\widehat{\gamma}_2$?

In other words, if shrinkage is applied cleverly, does it always work better for more efficient estimators?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
+25
$\begingroup$

Let me suggest an admittedly slightly boring counterexample. Say that $\hat{\beta}_1$ is not just asymptotically more efficient than $\hat{\beta}_2$, but also attains the Cramer Rao Lower Bound. A clever shrinkage technique for $\hat{\beta}_2$ would be: $$ \hat{\beta}_2^\ast = w \hat{\beta}_2 + (1 - w) \hat{\beta}_1 $$ with $w\in(0,1)$. The asymptotic variance of $\hat{\beta}_2^\ast$ is $$ V^\ast = \mathbb{Avar}(w \hat{\beta}_2 + (1 - w) \hat{\beta}_1) = \mathbb{Avar}(w (\hat{\beta}_2 - \hat{\beta}_1) + \hat{\beta}_1 ) = V_1 + w^2 (V_2 - V_1) $$ where the last equality uses the Lemma in Hausman's paper. We have $$ V_2 - V^\ast = V_2(1-w^2) - V_1(1-w^2) \geq 0 $$ so there is an asymptotic risk improvement (there are no bias terms). So we found a shrinkage technique that gives some asymptotic (and therefore hopefully finite sample) improvements over $\hat{\beta}_2$. Yet, there is no similar shrinkage estimator $\hat{\beta}_1^\ast$ that follows from this procedure.

The point here of course is that the shrinkage is done towards the efficient estimator and is therefore not applicable to the efficient estimator itself. This seems pretty obvious on a high level but I would guess that in a specific example this is not so obvious (MLE and Method of Moments estimator for the uniform distribution may be an example?).

Improve this answer
$\endgroup$
10
  • 1
    $\begingroup$ Thank you for the interesting example! (+1) However, it's not clear to me that this should be considered a counter example: it's both asymptotic and doesn't show that $\hat\beta_1$ can't be improved to have the same or lower risk. (In fact, your $\hat\beta_2^*$ automatically has, at best, the same risk as $\hat\beta_1$.) In order to provide a counterexample, the risk of a modified estimator $\hat\beta_2^*$ will have to be less than the risk of $\hat\beta_1$, and it's not clear that this is possible with this scheme. $\endgroup$
    – user795305
    Commented Jun 22, 2017 at 13:36
  • $\begingroup$ Thank you and point(s) taken. Let me however point out that nowhere in the question was it specified that the MSE of the modified $\hat{\beta}_2$ would need to be lower than that of $\hat{\beta}_1$. So $\hat{\beta}^\star_2$ is a valid shrinkage technique in this context. But I agree that this is just a partial answer and I am looking forward to seeing what other people have to say on this question. $\endgroup$
    – Matthias Schmidtblaicher
    Commented Jun 22, 2017 at 13:53
  • 1
    $\begingroup$ In the paragraph that begins "Suppose I have found...", the OP does seem to specify that. Am I misunderstanding? In what follows, let stars denote the modified estimators so that $\hat\beta_j^* = f_j(\hat\beta_j)$ for some (perhaps shrinkage) functions $f_j$. Suppose we find $\hat\beta_2^*$ so that $risk(\hat\beta_2) \ge risk(\hat\beta_2^*)$. In the referenced paragraph, OP asks if we can find some $f_1$ so that $risk(\hat\beta_1^*) \le risk(\hat\beta_2^*)$. $\endgroup$
    – user795305
    Commented Jun 22, 2017 at 14:54
  • $\begingroup$ I see. If this is the question, $f_1$ is simply be the identity and the answer is affirmative in the example. I read the question as "If we can find a function $f(\beta, x)$ so that $risk(f(\hat{\beta}_2,x)) < risk(\hat{\beta}_2) $, does there exist a $g(\beta, x)$ so that $risk(g(\hat{\beta}_1,x)) < risk(\hat{\beta}_1) $?" $\endgroup$
    – Matthias Schmidtblaicher
    Commented Jun 22, 2017 at 15:22
  • 1
    $\begingroup$ thanks for sharing these credits, even though I did not really answer your question... $\endgroup$
    – Matthias Schmidtblaicher
    Commented Jun 27, 2017 at 10:15
-2
$\begingroup$

This is an interesting question where I want to point out some highlights first.

  • Two estimators are consistent
  • $\hat{\beta}_1$ is more efficient than $\hat\beta_2$ since it achieves less variation
  • Loss functions are not the same
  • one shrinkage method is applied to one so that it reduces the variation that by itself ends up a better estimator
  • Question: In other words, if shrinkage is applied cleverly, does it always work better for more efficient estimators?

Fundamentally, it is possible to improve an estimator in a certain framework, such as unbiased class of estimators. However, as pointed out by you, different loss functions makes the situation difficult as one loss function may minimise quadratic loss and the other one minimises the entropy. Moreover, using the word "always" is very tricky since if one estimator is the best one in the class, you cannot claim any better estimator, logically speaking.

For a simple example (in the same framework), let two estimators, namely a Bridge (penalised regression with $l_p$ norm penalty) and Lasso (first norm penalised likelihood) and a sparse set of parameters namely $\beta$, a linear model $y=x\beta+e$, normality of error term,$e\sim N(0,\sigma^2<\infty)$, known $\sigma$, quadratic loss function (least square errors), and independency of covariates in $x$. Let choose $l_p$ for $p=3$ for the first estimator and $p=2$ for the second estimators. Then you can improve the estimators by choosing $p\rightarrow 1$ that ends up a better estimator with lower variance. Then in this example there is a chance of improving estimator.

So my answer to your question is yes, given you assume the same family of estimators and the same loss function as well as assumptions.

Improve this answer
$\endgroup$
8
  • $\begingroup$ it's not clear to me what you mean by take $p \to 1$. Given two estimators (say, from having $p=3$ and $p=2$ in $\ell_p$ regularization of least squares, like you discuss in your response), the question asks about ways to post-process these estimators (via, say, shrinkage). Specifically, it asks if there exists methods that can produce similar improvement (in terms of MSE) across consistent and asymptotically normal estimators. It's not clear to me what your answer is supposed to convey related to this. $\endgroup$
    – user795305
    Commented Jun 20, 2017 at 14:21
  • $\begingroup$ @Ben Thanks. the question is about shrinkage and I tried to take a simple example where applies shrinkage by imposing $l_p$ norm penalty on the estimator. I see it quite related. PS: LASSO ($l_1$ norm penalized likelihood) stands for Least Absolute Shrinkage and Selection Operator $\endgroup$
    – TPArrow
    Commented Jun 20, 2017 at 14:51
  • $\begingroup$ It's still not really clear to me. Are you proposing that we take the initial estimates $\hat\beta_1$ and $\hat\beta_2$ and then evaluate the $\ell_p$ proximal operator of them, so that the new estimates are $\hat\alpha^p_j = \arg\min_\alpha \|\alpha-\hat\beta_j\|_2^2 + \lambda \|\alpha\|_p$, for $j \in \{1,2\}$? If so, could you provide a proof (or some other argument) for your claims regarding MSE improvement? I tried to emphasize earlier that the question is asking about post-processing estimators--what exactly are your estimate for $p=2,3$ post processing? $\endgroup$
    – user795305
    Commented Jun 20, 2017 at 15:21
  • $\begingroup$ thanks @Ben, I feel we do not have a consensus in the definition of shrinkage. You take it like a post-process but me as an inline processing. I think we are both right since the question is not taking the type of shrinkage into account. PS: I guess what you mean from shrinkage is like hard-thresholding. $\endgroup$
    – TPArrow
    Commented Jun 21, 2017 at 11:59
  • $\begingroup$ Shrinkage can be both inline and as a post-processing. The examples you mentioned in your response are about "inline shrinkage", while the question asks about "post processing shrinkage". Notice that the question gives two estimators $\hat\beta_1$ and $\hat\beta_2$, then asks for a shrinkage technique to apply to $\hat\beta_1$ or $\hat\beta_2$. I think it might be worthwhile to reread the question in light of this. $\endgroup$
    – user795305
    Commented Jun 21, 2017 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.