1
$\begingroup$

I have just started to study Estimation theory and following the book "Fundamental of Statistical Signal Processing" by Steven M. Kay Vol 1. Most of the examples are based on the assumption that data follows a Gaussian distribution. COnsider, the example of estimating an unknown parameter $A$ from the relation: $x[n] = A + w[n]$ where w[n] is said to be $i.i.d$ random gaussian noise of known variance and zero mean. I have few conceptual questions and would appreciate intuitive notion behind them

Q1: Performing, $x[n]-w[n] = A$ would yield the unknown parameter A since w[n] is known. Then why does one need to consider the estimation using the estimator $\hat{A} = \frac{1}{N}\sum_{n=0}^{N-1}x[n]$ (Pg 12, Sec 2.3 Unbiased Estimators)

Q2: In general, does the number of samples, n play any role in estimation? Say n= 10 and n=1000, would this consideration affect the possibility of finding better estimates?

Q3: In general, why do we consider the samples and the noise to be $i.i.d$ ?? Is it ever possible that the samples are not iid? If so then when are they correlated?

Thank you

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Q1: you know the distribution of $w_n$, you don't know the value of $w_n$ for a give $n$. For a stupid example, you know that when you toss a coin $P(\text{head})=50\%$, but you don't know what the result of the next toss will be.

Q2: of course. $\hat{A}=\frac{1}{N}\sum_{n=1}^N x_n$, so its expected value is: $$E\left[\hat{A}\right]=E\left[\frac{1}{N}\sum_{n=1}^N x_n\right]=\frac{1}{N}\sum_{n=1}^N E[x_n]\overset{iid}{=}\frac{1}{N}\sum_{n=1}^N E[x]=\frac{1}{N}NE[x]=E[x]$$ but $$\text{Var}\left(\hat{A}\right)=\text{Var}\left(\frac{1}{N}\sum_{n=1}^N x_n\right)\overset{ind}{=}\frac{1}{N^2}\sum_{n=1}^N\text{Var}(x_n)\overset{id}{=}\frac{1}{N^2}N\text{Var}(x)=\frac{\text{Var}(x)}{N}$$ I.e. $\hat{A}$ is a good (unbiased) estimate, but its uncertainty (variance) decreases as $N$ increases. For another stupid example, if you get 2 heads in three tosses ($\hat{A}=2/3$) you can't say that the coin is unbalanced, but if you get 2000 heads in 3000 tosses ($\hat{A}$ is still $2/3$)... it's another matter!

Q3: because the iid assumption makes all simpler ;-) Samples may be not iid (e.g. in multistage sampling) and $x_n$ may be correlated to $A$. In such cases you have to use more complex estimation strategies, but learning the simpler ones at first is better.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your reply. Could you kindly elaborate the significance of iid? Why do we call independent and identically distributed? Under what circumstances or when is noise iid? What are the features for a signal to be labelled as iid? $\endgroup$
    – Ria George
    May 20, 2014 at 14:52
  • 1
    $\begingroup$ If each $w_n$ has a normal distribution with zero mean ($E[w_n]=0$ for all $n$) and the same variance ($\text{Var}(w_n)=\sigma^2$ for all $n$ - this is called homoskedasticity), then they are identically distributed. If they are independent, then $\text{Cov}(w_i,w_j)=0$ when $i\ne j$, i.e. they are not autocorrelated. Noise is tipically iid in experimental settings, when a measurement is repeated several times with the same instrument. If $A$ is not stochastic and $w_n$ is iid, then $y_n$ is iid because it's just a linear transformation of $w_n$. $\endgroup$
    – Sergio
    May 20, 2014 at 16:35
  • $\begingroup$ For the same question I had a doubt what if we need to estimate the variance of $w$ when $A$ is known to be rayleigh fading channel? $\endgroup$
    – undefined
    Sep 28, 2016 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.