I have just started to study Estimation theory and following the book "Fundamental of Statistical Signal Processing" by Steven M. Kay Vol 1. Most of the examples are based on the assumption that data follows a Gaussian distribution. COnsider, the example of estimating an unknown parameter $A$ from the relation: $x[n] = A + w[n]$ where w[n] is said to be $i.i.d$ random gaussian noise of known variance and zero mean. I have few conceptual questions and would appreciate intuitive notion behind them

Q1: Performing, $x[n]-w[n] = A$ would yield the unknown parameter A since w[n] is known. Then why does one need to consider the estimation using the estimator $\hat{A} = \frac{1}{N}\sum_{n=0}^{N-1}x[n]$ (Pg 12, Sec 2.3 Unbiased Estimators)

Q2: In general, does the number of samples, n play any role in estimation? Say n= 10 and n=1000, would this consideration affect the possibility of finding better estimates?

Q3: In general, why do we consider the samples and the noise to be $i.i.d$ ?? Is it ever possible that the samples are not iid? If so then when are they correlated?

Thank you


Q1: you know the distribution of $w_n$, you don't know the value of $w_n$ for a give $n$. For a stupid example, you know that when you toss a coin $P(\text{head})=50\%$, but you don't know what the result of the next toss will be.

Q2: of course. $\hat{A}=\frac{1}{N}\sum_{n=1}^N x_n$, so its expected value is: $$E\left[\hat{A}\right]=E\left[\frac{1}{N}\sum_{n=1}^N x_n\right]=\frac{1}{N}\sum_{n=1}^N E[x_n]\overset{iid}{=}\frac{1}{N}\sum_{n=1}^N E[x]=\frac{1}{N}NE[x]=E[x]$$ but $$\text{Var}\left(\hat{A}\right)=\text{Var}\left(\frac{1}{N}\sum_{n=1}^N x_n\right)\overset{ind}{=}\frac{1}{N^2}\sum_{n=1}^N\text{Var}(x_n)\overset{id}{=}\frac{1}{N^2}N\text{Var}(x)=\frac{\text{Var}(x)}{N}$$ I.e. $\hat{A}$ is a good (unbiased) estimate, but its uncertainty (variance) decreases as $N$ increases. For another stupid example, if you get 2 heads in three tosses ($\hat{A}=2/3$) you can't say that the coin is unbalanced, but if you get 2000 heads in 3000 tosses ($\hat{A}$ is still $2/3$)... it's another matter!

Q3: because the iid assumption makes all simpler ;-) Samples may be not iid (e.g. in multistage sampling) and $x_n$ may be correlated to $A$. In such cases you have to use more complex estimation strategies, but learning the simpler ones at first is better.

  • $\begingroup$ Thank you for your reply. Could you kindly elaborate the significance of iid? Why do we call independent and identically distributed? Under what circumstances or when is noise iid? What are the features for a signal to be labelled as iid? $\endgroup$ – Ria George May 20 '14 at 14:52
  • 1
    $\begingroup$ If each $w_n$ has a normal distribution with zero mean ($E[w_n]=0$ for all $n$) and the same variance ($\text{Var}(w_n)=\sigma^2$ for all $n$ - this is called homoskedasticity), then they are identically distributed. If they are independent, then $\text{Cov}(w_i,w_j)=0$ when $i\ne j$, i.e. they are not autocorrelated. Noise is tipically iid in experimental settings, when a measurement is repeated several times with the same instrument. If $A$ is not stochastic and $w_n$ is iid, then $y_n$ is iid because it's just a linear transformation of $w_n$. $\endgroup$ – Sergio May 20 '14 at 16:35
  • $\begingroup$ For the same question I had a doubt what if we need to estimate the variance of $w$ when $A$ is known to be rayleigh fading channel? $\endgroup$ – undefined Sep 28 '16 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.