2
$\begingroup$

I am wondering whether there is a general recipe for finding unbiased and consistent estimators of some non-random quantity.

For concreteness, I will discuss only discrete probability distributions with a finite support, but I would appreciate answers also for the continuous case.

Suppose that there is a discrete distribution over $M$ values:

$p_1=P(x_1)\\p_2=P(x_2)\\...\\p_M=P(x_M)$

Denote $x=(x_1,x_2,...,x_M)$ and $p=(p_1,p_2,...,p_M)$. The $x_i$ values are all different. The $p_i$ values are all non-negative and add up to $1$.

Any quantity that we may want to estimate can be written as $g(x,p)$ for some function $g$.

I am assuming that we know the function $g$ exactly, and that we know the value of $M$, but we do not know the values of $x$ and $p$.

So, given the function $g$ and given $M$:

  • Is there a general method that tells us whether an unbiased estimator for $g(x,p)$ exists?
  • Is there a general method that can find an unbiased estimator for $g(x,p)$, whenever it exists?
  • Is there a general method that tells us whether a consistent estimator for $g(x,p)$ exists?
  • Is there a general method that can find a consistent estimator for $g(x,p)$, whenever it exists?

(It may be the case that, given the function $g$ and given $M$, an unbiased or consistent estimator can be found only provided that $x$ and $p$ belong to some subset of values. In this case, I also wonder about whether there are methods to find this set).


For the sake of completeness, let me provide some definitions:

  • An estimator is a function $f(y_1,y_2,...)$ that takes as input a finite sequence of numbers $y_1,y_2,...\in \mathbb{R}$, and returns a value from $\mathbb{R}$. ($f$ does not depend on $p$ or $x$ defined above)

  • The estimator is said to be an unbiased estimator of $g(x,p)$ (defined above) (for a specific $N$), if for all $x$ and $p$, the expected value of $f(Y_1,Y_2,...,Y_N)$ equals $g(x,p)$, where $Y_1,Y_2,...,Y_N$ are independent random variables whose distribution is defined by $x$ and $p$ (see above).

  • (informal) The estimator is said to be a consistent estimator of $g(x,p)$ (defined above), if for all $x$ and $p$, in the limit of large $N$, the expected value of $f(Y_1,Y_2,...,Y_N)$ has limit $g(x,p)$, where $Y_1,Y_2,...,Y_N$ are independent random variables whose distribution is defined by $x$ and $p$ (see above).

$\endgroup$
15
  • 1
    $\begingroup$ On this forum, you needn't give definintions of estimators, unbiasedness, or consistency; these are all well-known concepts among the practitioners here. (And you certainly shouldn't give definitions that are wrong, or are at cross-purposes to the notation in your question.) $\endgroup$ – Ben Dec 27 '20 at 1:36
  • $\begingroup$ Of course they are well known. But I wanted to reduce ambiguity. For example, I wanted to make it clear that I'm talking about the IID case (sure, this is the standard case, but by no means is it the only case). Also, there are some subtleties involved here. For example, there could be a known unbiased estimator for some parameter given that we know the parametric family of the distribution. But my definitions made it clear that I am do not want to assume a parametric family ("for all x and p..."). Did I give a wrong definition anywhere? $\endgroup$ – Lior Dec 27 '20 at 1:54
  • $\begingroup$ If you want to add assumptions to your question, it is best to add them explicitly rather than trying to build them into "definitions" of statistical concepts. For example, your definition of the estimator uses a multiset input and therefore ignores the order of the input random variables, which is not the proper definition. If you want to specify that your estimator is invariant to permutations of the inputs, just say that explicitly, rather than giving a false definition of what an estimator is. $\endgroup$ – Ben Dec 27 '20 at 1:59
  • $\begingroup$ Ok, thanks for the comment. I'll take it into consideration next time. $\endgroup$ – Lior Dec 27 '20 at 2:00
  • 1
    $\begingroup$ You should do a search on this site for answers dealing with unbiased and consistent estimators, and ask more specific questions on this site if none of them answer those questions. $\endgroup$ – Peter O. Dec 27 '20 at 19:20
5
$\begingroup$

Let $D$ be the distribution determined by $x$ and $p$, and let $\phi$ be the mean of $(g(x, p)_0, g(x, p)_1, ..., g(x, p)_N)$, where $g(x, p)_i$ are i.i.d. random variables distributed as $g(D)$.

The answer to your first question is given in Halmos (1946): An unbiased estimator for $\phi$ exists if and only if $\phi$ is a homogeneous function over $D$ of degree at most $N$. In particular, if $M$ is 2 and $x$ is {0, 1} (so that the distribution is Bernoulli), then the only functions for which an unbiased estimator exists are polynomials of degree at most $N$ (Wästlund 1999, Goyal and Sigman 2012; see also this answer).

However, if $N$ is unlimited and $M$ is 2, then $g$ has an unbiased estimator in some subset of (0, 1) if and only if $g$ is continuous and is identically 0, identically 1, or polynomially bounded away from 0 and 1 in that subset (Keane and O'Brien 1994, Łatuszyński et al. 2009/2011).

I am not aware of any results that show whether a consistent estimator exists for a function if and only if an unbiased estimator exists for that function.

For more general distributions, Jacob and Thiery (2015) investigates the following question: Given a sequence of unbiased estimators, can we turn that sequence into an unbiased estimator of $g$ that is nonnegative almost surely? Without knowledge on what values $x$ can take on, the answer is no. However, if upper and lower bounds on $x$ are known, for example, this is possible for certain functions $g$; see the paper for details. See also Glynn (2016).

REFERENCES:

  • Wästlund, J., "Functions arising by coin flipping", 1999.
  • Goyal, V. and Sigman, K., 2012. On simulating a class of Bernstein polynomials. ACM Transactions on Modeling and Computer Simulation (TOMACS), 22(2), pp.1-5.
  • Keane, M. S., and O'Brien, G. L., "A Bernoulli factory", ACM Transactions on Modeling and Computer Simulation 4(2), 1994.
  • Jacob, P.E., Thiery, A.H., "On nonnegative unbiased estimators", Ann. Statist., Volume 43, Number 2 (2015), 769-784.
  • Glynn, P.W., "Exact simulation vs exact estimation", Proceedings of the 2016 Winter Simulation Conference, 2016.
  • Halmos, P.R., "The theory of unbiased estimation", Annals of Mathematical Statistics 17(1), 1946.
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.