# Effect of noise on an MLE estimate

Question : In general, what is the effect of noise on an estimate obtained from Maximum Likelihood Estimation technique?

• Could you provide more details? I think I grasped your problem, but I need more information. First question: is this measure, $M$, a scalar or a vector (if it is a vector, please provide a dimension for it, it may be needed in answering your question) ? Secondly, do you know if the MLE of $M$ is biased/unbiased/asymptotically unbiased? – PseudoRandom Jul 9 '14 at 9:12
• "Asymptotically unbiased" means the estimator is biased, however, its bias depends on some parameter (which we can control), say $K$. If $K \to +\infty$, then the bias tends to zero. Clearly, it's better to be unbiased than asymptotically unbiased (which is weaker). Trivial example: $\hat{x} = x + (1/K)n$, where $n \sim \mathcal{N}(\mu, \sigma^2)$, is asymptotically unbiased. I will post another comment with other questions regarding your problem. – PseudoRandom Jul 10 '14 at 9:04
• Ok, since I am not well-versed in AR, I'm still missing the general procedure. However, I can help on the MLE part. Your objective is to obtain an estimate of $(a,b)$, is that correct? Denoting the unbiased ML estimate of the measure with $\hat{x}$, you fed the system with two parameters: SNR and $\hat{x}$. Fix SNR=0 dB. The problem is: I don't see any direct dependency on $x$ in the AR formula. It is hidden in $y$ ? I mean: is $y$ actually a function of $x$ ? If so, is it a known function? – PseudoRandom Jul 10 '14 at 9:29
• (A) In general, the ML estimate of $x$ only implies that $\hat{x}_{ML}$ maximizes the likelihood function (i.e., the probability density function of the given problem). (B) The key point here is that $\hat{x}_{ML}$ is a continuous random variable, thus: if you simulate the system one time and obtain $\hat{x}_1$ and if you simulate the system a second time and get $\hat{x}_2$, $\hat{x}_1 = \hat{x}_2$ happens wp0 (with probability zero). Using the language of the article, $k_{ML}$ will always be different everytime you simulate the system. Different SNR alone is more than enough. – PseudoRandom Jul 11 '14 at 9:34
• Exactly, yes. Different SNR alone (with the same guessed parameters) is more than enough to get different estimates (i.e., realizations of the estimator). Can I post this last comments as an answer? – PseudoRandom Jul 11 '14 at 14:11

For future reference, the paper related to this question is:

Jaap C. Schouten, Floris Takens, and Cor M. van den Bleek, "Maximum-likelihood estimation of the entropy of an attractor", Phys. Rev. E 49, 126 – Published 1 January 1994.

In this paper, the authors want to obtain the Maximum Likelihood (ML) estimate of the Kolmogorov entropy, denoted with $K$. In order to do so, as a preliminary step, a quantity called $k=K \tau_s$, where $\tau_s = 1/f_s$ and $f_s$ is the sampling frequency, is estimated via ML techniques. This answer will focus on the statistical aspects of the question.

In this case, the pdf (probability density function) is given by eq. (17) of the paper, i.e. $$p(b_1,b_2,\dots,b_M; k) = (e^k -1)^M \exp \left\{ -k \sum_{i=1}^M b_i \right\}$$ where $M$ indipendent realizations of $b \in \mathbb{N}_0$, which is a random variable (RV) whose pdf is a geometric distribution, are observed. Reformulating it directly in function of $K$, the actual problem to be solved is $$\hat{K}_{ML} = \arg \max_{K} \ (e^{K \tau_s} -1)^M \exp \left\{ -K \tau_s \sum_{i=1}^M b_i \right\}$$ and the closed-form solution is given by eq. (20), which can be rewritten as $$\hat{K}_{ML} = - f_s \left| 1 - \frac{1}{\bar{b}} \right|$$ where $\bar{b}$ is the sample mean of the $b_i$. Observe that, since the $b_i$ are RVs, $\bar{b}$ is itself a RV. Clearly, $\hat{K}_{ML}$ maximizes the likelihood function. $\hat{K}_{ML}$ is an estimator, thus, by definition, it is a random variable. More precisely, its randomness comes from the fact that $\bar{b}$ is random. In principle, if the distribution of $\bar{b}$ could be calculated, then also the distribution of $\hat{K}_{ML}$ could be derived.

In general, if an estimator $\hat{x}$ is an absolutely continuous RV, denoting two estimates (i.e., realizations of the estimator) with $\hat{x}_1$ and $\hat{x}_2$, the condition $\hat{x}_1 = \hat{x}_2$ happens wp0 (with probability zero). This does not mean it can never happen, but only that it is "negligible" (using measure theory, a clear analogy is a zero-measure set). In fact, an event with probability zero is called negligible (and not "impossible") and this is the true subtility.

This does lead to somewhat counter-intuitive results, such as the fact that negligible events can happen even an infinite amount of times (!), given an infinitely long observation period.

SNR influences the behaviour of the system and plays an important role. I will explain it with an analogy. Consider the following simplified statement $$(1) \quad \hat{K}_{ML} \sim \mathcal{N}\left( K, \frac{1}{SNR} \right)$$ If SNR changes, the parameters of the distribution of $\hat{K}_{ML}$ change too and this directly affects the estimates. To see this more clearly, note that $$(2) \quad \lim_{SNR \to +\infty} \hat{K}_{ML} = K$$ exactly as expected. Equivalently, if you observe that $$(3) \quad \hat{K}_{ML} = K + n, \quad n \sim \mathcal{N}(0, \sigma^2)$$ where $\sigma^2$ is related to SNR, the gaussian distribution degenerates in the limit $\sigma^2 \to 0$ and $n$ collapses in its mean, i.e. zero. This is a very simple example: however, many important practical problems exhibit similar behaviour (and one should be wary of problems that do not!).

Summarizing, everytime the system is simulated, you obtain an estimate. This will be "almost surely" different from previous and future estimates, since every change in the parameters which govern the distribution of the estimator $\hat{K}_{ML}$ (be it SNR, guessed parameter, $M$, $f_s$, ...) directly affect the estimates. This should not be thought as a negative behaviour, since what actually matters are the statistical properties of the estimator, such as being unbiased, consistent, efficient, etc..., and this is also the reason why bounds such as the CRLB are useful and frequently used.

• Thank you for your answer.But I could not follow the technical term under (3) the gaussian distribution degenerates in the limit σ2→0 and n collapses in its mean, i.e. zero. Can you please explain what is meant by degenerate and collapse of mean? – Ria George Jul 11 '14 at 23:31
• In the limit $\sigma^2 \to 0$, a gaussian distribution $\mathcal{N}(\mu, \sigma^2)$ becomes a Dirac's Delta centered on its mean, $\mu$. This actually means that it is not a RV anymore: in fact, it becomes a deterministic term, i.e. $\mu$. This is what the collapse/degeneration actually means: any randomness disappears. – PseudoRandom Jul 12 '14 at 10:58