# Consistency of $f( x| \theta ) = \exp(-(x- \theta ))$

Prove that the second smallest observation in a random sample of size n from following pdf is consistent estimator of $$\theta$$

$$f( x| \theta ) = \exp(-(x- \theta )) , \qquad x > \theta$$

I just need a hint on how to start this question, because I dont have any idea on how to and where to start on this.

With the hints provided, I tried it further:

$$P( |X_{(2)} - \theta | > \ e ) < \dfrac{\operatorname{Var}(X_{(2)})}{\ e^{2}}$$ is the required condition for consistency

Using order statistics, the pdf of $$\ X_{(2)} = \dfrac{n(n-1)[1- \ exp(-(x- \theta ))]\ exp(-(x- \theta ))^{n-1}}{2}$$

Now I need to find the $$\operatorname{Var}(X_{(2)})$$

Which I am struck at, need a little assistance!

• You could start with the definition of consistent estimator. The second smallest observation is greater than the smallest observation which in turn is greater than $\theta$ so both would be biased upwards and the second smallest is more biased; but you need to show that this does not matter if they get close enough to $\theta$ (as required by the definition of consistent estimator) as the sample size increases. Jul 12, 2021 at 12:21
• Also note that the event $|X_{(2)}-\theta|>\epsilon$ is the same event that at most one of the $n$ observations are smaller than $\theta+\epsilon$ so this gives you a way of deriving an expression for the corresponding probability. Then take the limit. Jul 12, 2021 at 13:09
• Maybe one more hint will help you think about this effectively: show that for any whole number $r,$ the $r^\text{th}$ smallest observation is a consistent estimator of $\theta.$ In fact, see whether you can prove this for any location-family distribution whose support is bounded below. The generality of this statement might force you to think in terms of the concepts rather than just applying theorems or formulas.
– whuber
Jul 12, 2021 at 18:13

### Apply the definitions.

An estimator is a definite mathematical rule $$t$$ to compute a value given any finite dataset $$(x_1,x_2,\ldots, x_n).$$

When datasets are obtained by sampling a distribution $$F,$$ they produce independent and identically distributed random samples $$(X_1,X_2,\ldots, X_n)$$ for any $$n\ge 1.$$

Consider a family of distributions $$\mathfrak F.$$ Let $$\theta$$ be some property of these distributions. All that means is for any $$F\in\mathfrak F,$$ $$\theta(F)$$ is a definite number.

An estimator $$t$$ is consistent at $$F$$ for a property $$\theta$$ when $$t(X_1,X_2,\ldots,X_n)$$ converges to $$\theta$$ in probability. (This means the chance that $$t(X_1,\ldots, X_n)$$ differs from $$\theta$$ by any arbitrarily small but positive amount will eventually approach zero as $$n$$ grows large.)

An estimator $$t$$ is consistent at the family $$\mathfrak F$$ for a property $$\theta$$ when it is consistent at all $$F\in\mathfrak F$$ for $$\theta.$$

This all means you can employ a consistent estimator to obtain a reasonable guess of the value of $$\theta$$ even when you don't know which $$F\in\mathfrak{F}$$ is generating the samples.

### Reformulate the question.

Suppose every $$F\in\mathfrak F$$ has a greatest lower bound. A lower bound is some finite number $$\theta(F)$$ (depending perhaps on $$F$$) for which $$\Pr(X \le \theta(F)) = 0$$ when $$X$$ has $$F$$ for its distribution. $$\theta$$ is a greatest lower bound when, in addition to being a lower bound, it's always the case that $$\Pr(X \ge \theta(F)) \gt 0.$$

The exponential family in the question is a good example of this condition: it (implicitly) assigns zero probability to any values less than $$\theta$$ while for any $$x\gt \theta$$ the chance it assigns to values less than or equal to $$x$$ is given by $$1-\exp(\theta-x),$$ which is positive.

Here is the essence of the problem:

In any sample $$\mathbf X = (X_1,\ldots, X_n),$$ let $$t_r(\mathbf X)$$ be the $$r^\text{th}$$ smallest value among the $$X_i$$ (where $$r$$ is any fixed whole number). Prove that $$t_r$$ consistently estimates the greatest lower bound $$\theta.$$

### The solution follows immediately.

According to the definitions, we need to show that for any $$F\in\mathfrak F,$$ the $$r^\text{th}$$ smallest value in a sample of size $$n$$ is highly likely to be very close to the greatest lower bound $$\theta(F).$$

The idea should now be clear: by choosing $$n$$ very large, $$r$$ will eventually be smaller than any specified fraction of $$n,$$ say $$n\delta.$$ When $$\delta$$ is tiny, the $$n\delta$$ quantile of $$F$$ (no matter what $$F$$ might be) must be close to the greatest lower bound $$\theta(F).$$ Thus, the $$r^\text{th}$$ smallest value of a sample consistently estimates $$\theta,$$ QED.

If you need to see the details, here they are spelled out.

Let $$\epsilon \gt 0$$ indicate just how close you want $$t_r$$ to get to $$\theta.$$ Let $$n$$ be any sufficiently large possible sample size (so we may assume $$n\ge r$$). Define $$p(\epsilon; n, r, F) = F(\theta(F)+\epsilon).$$ This is relevant because $$q=1-p$$ is the chance that any single value drawn from $$F$$ exceeds $$\theta(F)+\epsilon.$$ (For convenience I will drop the "$$(\epsilon;n,r,F)$$" after $$p$$ and $$q.$$)

Because $$\theta$$ is a greatest lower bound, $$p \gt 0$$ and therefore $$q \lt 1.$$

Let $$\mathbf{X}_n$$ be a sample of size $$n$$ from $$F.$$ Consider the chance that $$t_r(\mathbf{X}_n)$$ is close enough to $$\theta(F).$$ That is, let's contemplate the chance that the $$r^\text{th}$$ smallest sample value does not exceed $$\theta(F)+\epsilon.$$ Put another way, at most $$n-r$$ of the sample values may exceed $$\theta(F)+\epsilon.$$

We can estimate this chance.

Each of the $$X_i$$ individually has a chance $$q$$ to exceed the threshold (because they all have $$F$$ for their distribution). And since the $$X_i$$ are independent, the chance that $$n-r$$ or fewer of them are this large is given by the Binomial$$(n,q)$$ distribution evaluated at $$n-r.$$

We know the mean of this Binomial distribution is $$nq$$ and its variance is $$nq(1-q).$$ Thus, $$n-r$$ differs from the Binomial mean by

$$\frac{(n-r) - nq}{\sqrt{nq(1-q)}} = \sqrt{n}\frac{(1-q) - r/n}{\sqrt{q(1-q)}}$$

standard deviations. Chebyshev's Inequality asserts the chance of a deviation from the mean of this magnitude (or greater) can be no larger than the reciprocal square of this value,

\begin{aligned} \Pr(t_r(\mathbf{X}_n)\le\theta(F)+\epsilon) &\le \left(\sqrt{n}\frac{(1-q) - r/n}{\sqrt{q(1-q)}}\right)^{-2} \\ &= \frac{1}{n}\frac{q(1-q)}{(1-q-r/n)^2} \\ &\le \frac{1}{n}\frac{q}{1-q}\left[1 + \frac{2}{1-q}\right]. \end{aligned}

Clearly, no matter what $$0\lt q\lt 1$$ might be and no matter how large $$r$$ might be, the limiting value of this expression as $$n$$ grows large is zero.

Obviously there is no chance that $$t_r(\mathbf{X}_n)$$ is less than $$\theta(F),$$ because this estimator always equals one of the $$X_i$$ and since $$\theta$$ is a lower bound, none of the $$X_i$$ have any chance of being this small. We have thereby shown that as $$n$$ grows large, $$t_r$$ is squeezed between $$\theta(F)$$ and $$\theta(F)+\epsilon$$ with arbitrarily high probability, no matter how small (but positive) $$\epsilon$$ might be.

This completes the proof that $$t_r(\mathbf{X}_n)$$ converges to $$\theta(F)$$ in probability for any $$F\in\mathfrak F,$$ and that's what it means for $$t_r$$ to be a consistent estimator.

• I understood the solution, but I had one doubt , that how to process such information in exam , suppose if I am in exam then how do I know I have to think like this ? Or if you could give a tip on it ( on general basis in inference) because I am facing doubt in every third question of inference?? Jul 15, 2021 at 14:43
• Practice and reflect. Aim to answer homework questions and exercises by recognizing and applying concepts. Don't apply any formulas you couldn't sit down and derive on the spot. (And do that: every time you use a new formula, re-derive it until you can do that instantly.) Don't be satisfied with just getting answers: return to your work from time to time and consider how you could improve on it. Study answers produced by others. That experience will translate to finding better solutions more quickly, whether on exams or with real-life problems.
– whuber
Jul 15, 2021 at 14:46
• I should also add: memorize definitions. Once you are familiar with the definitions, your entire answer can be derived as shown in the two paragraphs below "Solution." The rest is a matter of technique, not insight.
– whuber
Jul 15, 2021 at 14:48
• thank you sir , I will try to focus more concept than on just answer Jul 15, 2021 at 14:56