In this post, I stated that the loss function criterion does not determine an frequentist estimator (though it determines a Bayesian estimator).

However, I am now not so sure anymore, so I'm wondering whether we can somehow pinpoint an estimator based solely on a loss function. I know that if we first constrain the class of estimators (e.g. by requiring unbiasedness) then we can do this, but I am wondering if it is possible to do it solely on the basis of a loss function (i.e. without adding the requirement of e.g. unbiasedness which does not follow from the loss function).

I write down here a sequence of failed attempts:

Assume that we have a parameter $x$, and a random variable $Y$, with realized data $y$. Assume also that we have a loss function $L$ that we want to minimize.

The attempt in that earlier post, is to state:

$$ \hat x(\,. ) = \text{argmin} \; \mathbb{E} \left( L(x-\hat x(Y)) \; | \; x \right)$$

First attempt. My critique was that the solution to this problem is trivially $\hat x(\cdot )=x$. But of course, since we don't know $x$, this cannot be actually applied, and my informal/vague conclusion was that maybe there is no solution to this problem.

However, this assumes that we are minimizing w.r.t. a variable.

Second Attempt. If we instead demand that the estimator $\hat x(\cdot)$ is a function from the space of $Y$ to $\mathbb R$, then we demand this estimator will have to be the same function, regardless of the value of $x$. The problem with this is, that the loss function is a function of a real value, not of a function. That means that the problem is undefined, since the function that minimizes this expectation for $x=5$, is not the same function that minimizes it if $x=3$. The loss function does not give a criterion on how to pick between those multiple values.

So essentially we need a criterion that says: "regardless of what $x$ is, the function should minimize the expected loss function, weighted by some weight $w(x)$ on the value of $x$"

Third Attempt. So we require that the sum of the expected loss conditional on $x$, weighted by some weight function $w(x)$ is minimized: $$\text{argmin }\int \mathbb E(L(x-\hat x(Y))|x)\cdot w(x)dx$$

Note however, that if we interprete the weight function $w(x)$ as the prior on $x$, then this is simply, the zero-information (Bayesian) expectation of the conditional Expected loss function:

$$\text{argmin }\mathbb E(\mathbb E(L(x-\hat x(Y))|x)$$

Which by the law of iterated expectations is equal to

$$\text{argmin }\mathbb E(L(x-\hat x(Y))$$

Which is essentially minimizing the Bayesian loss function, when no sample has yet been made. And the problem with this is that it is no longer a frequentist estimator, since we have invoked the (arbitrary from a frequentist perspective) $w(x)$ function, or prior $p(x)$.

So we need some way to formulate a criterion w.r.t. a loss function that determines a unique estimator, without referring to arbitrary weight functions

Fourth attempt?

Is there a way to do this?

  • $\begingroup$ With no further constraint there is no unique solution. The fact is that the constant estimator $\hat{x}(y)=x_0$ minimises the loss when $x=x_0$, a solution that cannot be dominated by a non-constant $\hat{x}$. $\endgroup$ – Xi'an Feb 11 '18 at 15:49
  • $\begingroup$ @Xi'an, I already addressed that, but yes you are right of course. $\endgroup$ – user56834 Feb 11 '18 at 16:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.