Consider a sequence of continuous random variables $\{X_n\}^\infty_{n=1}$ that are independent and identically distributed under the probability density function $f_\theta (x)$, where $\theta \in [\ell ,u]$ is an unknown but deterministic parameter, $\ell < 0 <u$ and $\theta \neq 0$.Consider the random sequence:

\begin{equation} S_n = \sum_{i=2}^n \log\frac{f_{\hat{\theta}_{1:i-1}}(X_i)}{f_0(X_i)} \end{equation}

where $\hat{\theta}_{1:i-1}$ is the maximum likelihood estimate of the parameter $\theta$ using the observations $\{X_n\}^{i-1}_{n=1}$, and $S_0 \triangleq S_1 \triangleq 0$. Let $ \tau =\inf\{n \geq 2 : S_n \leq 0\}$. I am trying to show that $S_n$ may diverge to infinity before becoming non-positive, with non-zero probability, i.e., \begin{equation} \mathbb{P}(\tau = \infty) >0. \end{equation} My intuition says that this should hold due to the fact that $\hat{\theta}_{1:i-1}$ converges to $\theta$ almost surely as $i \rightarrow \infty$. Do you think this result should hold for the case that the maximum likelihood estimator is used to estimate the parameter $\theta$. I would really appreciate if anyone could provide any results that could be useful to show this claim! Thanks!

  • $\begingroup$ Agreed with your intuition: for large $n$ we have $S_{n+1} \approx S_n + \xi_{n+1}$ where $\xi_n = \frac{\ln f_\theta(X_{n}) }{f_{0}(X_{n})}$ is independent of everything preceding in the process with $\mathbb{E}\xi_n > 0$. So it is eventually like a random walk with positive drift. However $\theta = 0$ is more problematic..... in fact I can't imagine anything other than $\mathbb{P}(\tau = \infty) = 0$ for that case $\endgroup$ May 22, 2018 at 21:39
  • $\begingroup$ It is assumed that $\theta \neq 0$ for this problem. I am not sure if any stronger assumptions should be made. The analysis here seem more difficult since we have non i.i.d. and a lot of tools can not be used. However I was wondering whether the claim could be shown for the case that $f_\theta$ belongs to an exponential family, or is Gaussian. $\endgroup$ May 22, 2018 at 22:01
  • $\begingroup$ Oh I missed the $\theta \neq 0$ assumption. As for other assumptions it would be helpful to assume you have enough regularity for the MLE to converge. Anyway, in the argument I was outlining you approximately lower bound $S_n$ by i.i.d increments for large $n$, say $n > N$. Then you combine that with a separate argument to say $S_N > 0$ with positive probability. $\endgroup$ May 22, 2018 at 22:43
  • $\begingroup$ I'll try to explain in more detail. Choose $\epsilon > 0$ small enough such that $\mathbb{E}\inf F(\theta';X_n) > 0$ where the inf is taken over $\theta' \in [\theta-\epsilon,\theta + \epsilon]$ and $F(\theta',x) = \ln f_{\theta'}(x) / \ln f_0(x)$. You have $\hat \theta_n \to \theta$, so eventually $\hat\theta_n \in [\theta-\epsilon,\theta+\epsilon]$ and your increments are bounded below by i.i.d variables. $\endgroup$ May 22, 2018 at 22:47
  • $\begingroup$ I'm not sure I understand what you mean by " bounded below by i.i.d variables". Which are those random variables and why are they a bound for the increments? Also, isn't that statement you are talking about asymptotic? I can not see the connection with the statement to be proven. Thank you so much for your time! I really appreciate the help! $\endgroup$ May 23, 2018 at 0:37

1 Answer 1


The following is more of an extended comment with explicit calculations for the Gaussian case to illustrate the idea.

Consider the unit variance case so I can avoid typing $\sigma$, denote the parameter of interest (mean) by $\mu > 0$ for both laziness and aesthetics. Thus we are interested in $$ f_\mu(x) = c\exp(-(x-\mu)^2/2) $$ where $c$ is the corresponding normalising constant.

Continue to let $X_1,\ldots$ be your random sample from $f_\mu$. The MLE for $\mu$ is just the sample mean, $$ \hat \mu_n = \frac{1}{n}\sum_{i=1}^n X_i. $$

Take any $\epsilon > 0$. The SLLN implies $$ \mathbb{P}(| \hat \mu_n - \mu| < \epsilon, n > N) \to 0 $$ as $N\to \infty$. So for $N$ large enough we can assume $| \hat \mu_n - \mu| < \epsilon$ for $n > N$ off an event with arbitrarily small probability.

Observe $$ \ln f_m(x)/f_0(x) = \ln\exp(-(x-m)^2/2 + x^2/2) = m(2x-m)/2 $$

and $$ m(2x -m) > F(x) := (\mu \pm \epsilon)(2x - \mu - \epsilon) $$ for $|m - \mu| < \epsilon$ (where the $\pm$ is depending on whether $m-2x \ge 0$ or not).

So if $\epsilon < \mu$ then no matter the exact value of $m \in (\mu-\epsilon, \mu+\epsilon)$, $$ \ln f_m(X_n)/f_0(X_n) > F(X_n) $$ where the right hand side is a random variable with positive expectation.

In particular, on an event of arbitrarily high probability, $$ \ln f_{\hat \mu_n}(X_{n+1})/f_0(X_{n+1}) > F(X_{n+1}) $$ for all $n > N$ (by choosing fixed $N$ large enough).

Now, the RW $\tilde S_k := \sum_{i=1}^k F(X_{N+i})$, $k \ge 0$ (starting from 0) has i.i.d increments with positive expectation so explodes to infinity before hitting zero with positive probability not depending on $N$. So WLOG choose $N$ in the previous step large enough so that $| \hat \mu_n - \mu| < \epsilon$ for $n > N$ fails on an event with smaller probability. Thus, the intersection of $(\tilde S_k)$ exploding before going negative and $| \hat \mu_n - \mu| < \epsilon$ for $n > N$ has positive probability. But on the latter event $(S_{N+k} - S_N)$ is lower bounded by $\tilde S_k$ so also explodes before hitting zero.

Now, you just need to show that the probability that $S_1>0,S_2 > 0,\ldots, S_N > 0$ (finitely many variables) is positive (with $| \hat \mu_n - \mu| < \epsilon$ for $n > N$ still holding off an event with arbitrarily small probability), which seems straightforwards.

  • $\begingroup$ Thanks for the answer! I will try to see if a similar proof works for the exponential family case. $\endgroup$ May 25, 2018 at 21:34
  • $\begingroup$ So as we said before there is a problem when $\mu =0$. In the proof you also have to be able to set your $\epsilon$ to be smaller than $\mu$. Since $\mu$ can go arbitrary close to $0$, doesn't that mean that for those cases $\epsilon$ will be very close to $0$. Will that lead to $N$ becoming unbounded, and if yes wouldn't that be a problem when proving the last simple argument? $\endgroup$ May 31, 2018 at 6:24
  • $\begingroup$ Yep the choice of $N$ and $\epsilon$ depends on $\mu$ but the latter is assumed constant, i.e. for the problem stated the argument works but indeed it's not uniform in $\mu$ might cause problems in practice? Btw I no longer think the "last simple part" is quite as easy as I initially thought, or rather it is but then you need to do the main part conditioned on $S_1,\ldots,S_n$ being positive. $\endgroup$ May 31, 2018 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.