6
$\begingroup$

Anyone got any suggestions? I tried finding an estimator explicitly and showing it had to be consistent, along with showing the consistency of the mle but neither seems to be working. Think I'm just missing a trick somewhere.

Let $X_i\sim\mathrm{Bernoulli}(1/2+\theta\cdot a_i)$ where $\left\{a_i\right\}_{i\geq 1}$ is a sequence of known positive constants such that $a_i$ decreases to 0. Show there is a consistent estimator for $\theta$ if and only if $\sum_{i=1}^{\infty}a_i ^2 =\infty$.

| cite | improve this question | |
$\endgroup$
  • 1
    $\begingroup$ Can you tell us the source of this problem? $\endgroup$ – Zen Jun 15 '17 at 22:33
  • 6
    $\begingroup$ Obtuse hint: As stated, I think (though didn't check) that this is clearly false. Let $U \sim \mathcal U[0,1]$. Take $X_i = 1_{(U \leq \theta a_i + 1/2)}$. The $X_i$ satisfy your problem statement. Now, assume $a_i$ decreases monotonically to zero. Then, your data is a sequence of the form $1,1,\ldots,1,0,\ldots$. This allows you to determine $\theta$ to lie with probability one inside a given interval. But, you get no further information about $\theta$. You can make the sequence of $a_i$ converge arbitrarily fast or slow within this setup and will never learn enough about $\theta$. $\endgroup$ – cardinal Jun 15 '17 at 23:18
  • 4
    $\begingroup$ (So...perhaps your problem statement is missing a crucial assumption. Given that assumption, how might you proceed? Try the "easy" direction first.) $\endgroup$ – cardinal Jun 15 '17 at 23:20
  • 1
    $\begingroup$ @cardinal there are no assumptions missing from how the question was stated where I came across it. I don't see what this missing assumption could be either. $\endgroup$ – Gosset's Student Jun 16 '17 at 12:54
1
$\begingroup$

Consider the simpler case where $X_i \sim N(\theta a_i,1)$, then the MLE $$\hat{\theta} = \sum_{i=1}^n a_iX_i/\sum_{i=1}^n a_i^2 \sim N(\theta, 1/\sum_{i=1}^n a_i^2).$$ Note that if $\sum_{i=1}^n a_i^2 $ diverges, then $\hat{\theta}$ is consistent since the variance goes to 0 and it is unbiased; if $\sum_{i=1}^n a_i^2 $ converges, $\hat{\theta}$ is asymptotically normal with constant variance, which implies inconsistency of MLE and there is no other estimator that can achieve consistency in this case.

Going back to your problem, I think you can show that asymptotic variance of the MLE, by computing the Fisher information, is the same order of $1/\sum_{i=1}^n a_i^2$. If $\sum_{i=1}^n a_i^2 < \infty$, it means the information is not accrued fast enough, MLE will not be consistent and there are no other consistent estimators. If $\sum_{i=1}^n a_i^2 = \infty$, MLE will be consistent.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The expected value of the MLE in this case is $$E(\hat{\theta}) = \frac {\sum_{i=1}^n a_iE(X_i)}{\sum_{i=1}^n a_i^2} = \frac {\sum_{i=1}^n a_i(1/2 + \theta a_i)}{\sum_{i=1}^n a_i^2} = \frac 12 \frac {\sum_{i=1}^n a_i}{\sum_{i=1}^n a_i^2} + \theta$$ so why do you say that it is unbiased? The first term is not equal to zero for any finite $n$ (since the $a_i$-sequence converges to zero). (CONT'D) $\endgroup$ – Alecos Papadopoulos Dec 16 '17 at 13:53
  • $\begingroup$ (CONT'D) Regarding consistency, we want $$P\left(\left|\frac {(1/n)\sum_{i=1}^n a_iX_i}{(1/n)\sum_{i=1}^n a_i^2} - \theta\right| < \epsilon\right) \to 1$$ It holds that the denominator goes to zero (irespective of whether $\sum_{i=1}^n a_i^2$ goes to infinity or not, so it is not at all clear that for consistency we need $\sum_{i=1}^n a_i^2 \to \infty$. I think you need to provide more proof for the assertions you are making. $\endgroup$ – Alecos Papadopoulos Dec 16 '17 at 13:53
  • $\begingroup$ 1. I did not say that is the MLE for the Bernoulli. I am considering the simpler case first, which is Normal. MLE is also normally distributed in this case, so why it is not unbiased? It is quite easy to show its the sufficient and necessary condition. The MLE in the Bernoulli problem here is not in closed form, but you compute its asymptotic variance. $\endgroup$ – Statisfun Dec 17 '17 at 2:54
1
$\begingroup$

Here is an estimator which can consistently estimate $\theta$ and for which the condition $\sum_{i=1}^{\infty}a_i ^2 =\infty$ is relevant.

We have

$$E(X_i) = \frac 12 + \theta a_i \implies \frac{2E(X_i)-1}{2a_i} = \theta$$

Set

$$Z_i = \frac{2X_i-1}{2a_i} \implies E(Z_i) = \theta$$

and

$$\text{Var}(Z_i) = \text{Var}(X_i/a_i) = \frac {1}{4a_i^2} -\theta^2$$

Consider the sample mean $\bar Z_n$. Its expected value is always $\theta$, and its variance is

$$\text{Var}(\bar Z_n) = \frac {1}{n^2}\sum_{i=1}^n\frac{1}{4a_i^2} -\frac{\theta^2}{n}$$

Ignoring weird situations, we want this variance to go to zero in order to make certain that $\bar Z_n$ will consistently estimate $\theta$. So we want

$$\frac {1}{n^2}\sum_{i=1}^n\frac{1}{a_i^2} \to 0 \implies \frac {n^2}{\sum_{i=1}^n\frac{1}{a_i^2}} \to \infty$$

Now from the Harmonic-Arithmetic mean inequality , we have that

$$\frac {n}{\sum_{i=1}^n\frac{1}{a_i^2}} < \frac 1n \sum_{i=1}^na_i^2 \implies \frac {n^2}{\sum_{i=1}^n\frac{1}{a_i^2}} < \sum_{i=1}^na_i^2$$

Since we want the left side of the inequality to go to infinity, it follows that $\sum_{i=1}^{\infty}a_i ^2 =\infty$ is a necessary condition for this to happen, i.e. a necessary condition for $\bar Z_n$ to consistently estimate $\theta$.

(Follows the original exploratory post).

We have that

$$\{a_i\} \to 0 \implies \frac 1 n \sum a_i \to 0 \tag{1}$$

and

$$\{a_i\} \to 0 \implies \{a_i^2\} \to 0\implies \frac 1 n \sum a_i^2 \to 0 \tag{2}$$

Now, the OP does not state it but I will assume that the rv's are independent, since the focus is elsewhere. Also, I note that the Bernoulli distribution belongs to the Exponential family and so the sum of observations is a sufficient statistic, it has all the information that the sample can give us.

More over, Chebychev's Law of Large Numbers applies (mean and variance of each $X_i$ is finite), namely, the sample average of the $X$s converges in probability to the limit of the average of their means, and we have

$$\text{plim}\bar X = \lim\frac 1n \sum E(X_i) = \frac 12 + \theta \lim\frac1n \sum a_i = \frac 12 \tag{2}$$

So as sample size tends to infinity, information on $\theta$ is eliminated if we use the sufficient statistic.

The same we would obtain if we looked at the variance. The variance of $X_i$ is

$$\text{Var}(X_i) = \left (\frac 12 + \theta a_i\right) \left(1-\frac 12 - \theta a_i\right) = \left (\frac 12 + \theta a_i\right) \left(\frac 12 - \theta a_i\right) = (1/4) - \theta^2 a_i^2$$

and calculating any sample mean will again tend to eliminate the information on $\theta$ as the sample size increases, due to $(2)$.

So the challenge here appears to be that we have to abandon the sufficient statistic.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Unless all the $a_i=0$, your "necessary condition" $(5)$ is vacuously true for all $n$--please check that it states what you intended. Condition $(4)$ is nonsensical because "$n$" cannot appear after "$\to$". Once your analysis states what you intended, then a closer examination might be worthwhile. $\endgroup$ – whuber♦ Nov 4 '17 at 22:09
  • $\begingroup$ @whuber I fixed $(4)$, but I don't understand your comment on $(5)$. It says that he limit of the average of the $a_i$s should be strictly positive. Why is this vacuously true? What if, as cardinal's comment noted, that $a_i$s become zero after some finite value of the index? Won't their average converge to zero then? $\endgroup$ – Alecos Papadopoulos Nov 4 '17 at 22:25
  • $\begingroup$ I think you're abusing notation. Did you mean to write something like $$\lim_{n\to\infty}\frac{1}{n}\sum_{i=1}^n a_i \gt 0?$$ $\endgroup$ – whuber♦ Nov 4 '17 at 22:26
  • $\begingroup$ @whuber Certainly. And ok, I wrote it the way you wrote it to be clear. $\endgroup$ – Alecos Papadopoulos Nov 4 '17 at 22:29
  • 1
    $\begingroup$ How can your necessary condition possibly hold when $a_i$ "decreases to $0$", as stipulated in the question? $\endgroup$ – whuber♦ Nov 4 '17 at 22:38
0
$\begingroup$

I think I have finally figured out a way to solve this problem. The details are yet to be worked out but I feel if the sum is finite then for two different $\theta_1,\theta_2$ the sequence of likelihood ratios will be mutually contiguous, which would imply there is no consistent estimator. A complete solution will be added when time permits.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.