0
$\begingroup$

Let's say that we make multiple noisy observation from a sensor node where $h$ is the parameter we want to deduce and $v$ is the noise.

$$y[k] = h + v , k=[0,1,..n] $$

Question: The PDF for each individual measurement shall be of the following form:

$$F(y(k)) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^\frac{(y(k)-h)^2}{2 \sigma^2}$$

Then for joint PDF why do we use the product of individual PDFs using the following formula assuming noise is i.i.d.? What is the purpose of product in laymen terms? Shouldn't it be summation and averaging of individual PDFs?

$$ \prod_{k=0}^n F(y(k)) $$.

The YouTube reference at 6:54

$\endgroup$
  • $\begingroup$ This applies only if the distributions are independent. The formula is simply a mathematical result based on the definition of a probability density. Why would you even think that summation would be appropriate? Summation does apply to the union of the probability of disjoint events. But that has nothing to do with this. $\endgroup$ – Michael Chernick Sep 8 at 15:00
0
$\begingroup$

Then for joint PDF why do we use the product of individual PDFs [...] assuming noise is i.i.d.? What is the purpose of product in laymen terms? Shouldn't it be summation and averaging of individual PDFs?

Let's not mix estimators to their associated PDFs.

An estimator $\hat{h}$ for $h$ takes this shape according to the intuitive averaging approach:

$$\hat{h} = \frac{1}{n}\sum_{k=1}^ny[k] = h + \frac{1}{n}\sum_{k=1}^nv[k]$$

Is this a good estimator? It depends on the noise. If we assume that different measurements of it are independent of each other, and all have zero-average, then we're reasonably sure that the estimator will not fail us. Then we also assume that the measurements are also identically distributed because it makes sense and makes calculations much easier. So the i.i.d. guarantees us that the above estimator is unbiased (i.e. its expected value is $h$).

Anyway, the estimator $\hat{h}$ is another random variable because it depends on the average of the different $v[k]$ that we get (for each sample of $n$ measures we will generally get different results). What is the PDF of this new random variable?

This is where the rules of composition for PDFs kick in. The different $v[k]$ are i.i.d., so we can have the joint PDF as the product of the respective $PDF$s (because of independence, see @Fr1's answer about this) and this product is quite easy to calculate (because they're all identical).

Anyway, this joint distribution is not the distribution of $\hat{h}$, because it still depends on each measurement individually considered a random variable itself. We have to take into account that $\hat{h}$ is the average of all of them, and do a few integrations for this. We end up with this formula:

$$ f_{\hat{h}}(x) = \frac{1}{\sqrt{2 \pi \sigma_{a}^2}} e^\frac{(x - h)^2}{2 \sigma_{a}^2} $$

i.e. this new variable resulting from averaging the samples is still following a normal distribution, with mean value $h$ (it's unbiased) and a different variance according to the following formula (this is where the identical assumption makes calculations easier):

$$\sigma_{a}^2 = \frac{\sigma^2}{n}$$

which means that the more measurements we take, the less the variance for this estimator will be. Hence, the estimator is both unbiased and in a sense good, because it gets better as we take more measurements.

(Arguably, it's not that good, in the sense that the standard deviation decreases like $\sqrt{n}$, which means that to half the standard deviation you have to take four times measurements).

So, to recap:

  • we make the assumption of independence so that the average is a good estimator of the parameter $h$;
  • averaging independent variables means multiplying their respective PDFs to obtain the joint distribution of all measurements, which is an $n$-dimensional function;
  • integrating this $n$-dimensional function under the constraint of taking an average of all variables yields the PDF of the estimator;
  • adding the "identically distributed" hypothesis further simplifies calculations and makes results easier to interpret.
$\endgroup$
  • $\begingroup$ This is quite holistic explanation for getting good insight onto how we determine parameter estimation (NOT just averaging out as I was thinking), you did grasp the very core of the confusion I was facing. In laymen terms we have a parameter we want to estimate but the measurement is noisy so we take several measurements and draw a joint PDF and by the "magic" of integration/summation we come up with the form that allows us to determine the "noise-free" estimate of the parameter. Did I sum this up correctly in laymen terms? $\endgroup$ – GENIVI-LEARNER Sep 8 at 21:10
  • $\begingroup$ Also is there any article you would suggest for reading that further highlights this and its limitations? $\endgroup$ – GENIVI-LEARNER Sep 8 at 21:10
  • $\begingroup$ I'd say your summary is correct as long as we define "noise-free" as "the estimation of what remains of the noise after the average is below an acceptable threshold and can be neglected". In that case, math shows that you can make the standard deviation (and so almost any confidence level you can think of) as little as you want if you're willing to invest in doing the right amount of measurements. I honestly can't think of any article or book discussing this... sorry. $\endgroup$ – polettix Sep 8 at 21:18
  • $\begingroup$ I would like to ask in general the estimator you wrote above is meant to take sample measurement from a single location (assuming we are doing spacial sampling), in which case we make assumption that the noise term is i.i.d. In stochastic process such as Gaussian process we are taking sample measurement from different location/s and add assumption that noise term is not i.i.d and correlated hence the use of co-variances. Is this correct understanding? $\endgroup$ – GENIVI-LEARNER Sep 9 at 15:27
  • $\begingroup$ I studied stochastic processes "a while ago" (25+ years :) and while your reasoning seems sound I can't honestly say it's correct. Additionally, I would say that the same reasoning might apply in time as well, so I don't see the need to introduce the "spatial" constraint on sampling. $\endgroup$ – polettix Sep 9 at 20:12
0
$\begingroup$

When the noises are assumed to be independent, then their joint density function is the product of marginal densities by definition of independence. See pag. 9 of this. You can see that the random variables A and B are independent if $P(A \cap B)=P(A)P(B)$. So here, the joint density is the product of marginal densities. Summation can’t be the solution as, intuitively, if you flip a coin twice, the prob that you will have two heads in a row (1/2*1/2=1/4) must be lower than the probability of having two generic results either head or tail (1/2+1/2=1). The average would just be the sum divided by two, so conceptually nothing changes and it is still wrong for the same reason. Indeed here you are seeking the intersection between having a tail at the first flip and a tail at the second. You are not happy with any two outcomes, you want exactly two heads in a row.

That is why you have a product.

Finally, the fact that the noises are identically distributed allows you to keep the same pdf function for each observation. That is why, in the product mentioned above, you have the same pdf function for each term of the product.

Which explains the shape of the joint pdf as a product of marginal pdf (due to the assumed independence of noises) with the same pdf function (due to the assumed identical distribution).

$\endgroup$
  • $\begingroup$ This makes a lots of sense and I got the intuition that the product will provide joint probability of two events such as two coin flips. My question is that what exactly are we seeking using joint PDF. Can you please elaborate this further "Indeed here you are seeking the intersection between having a tail at the first flip and a tail at the second. You are not happy with any two outcomes, you want exactly two heads in a row" $\endgroup$ – GENIVI-LEARNER Sep 8 at 14:18
  • $\begingroup$ You are seeking the probability that, given for simplicity two random variables A and B (where in the example they were the outcome of a coin flip repeated two times, so independent and identically distributed coin flip), A=outcome1 AND B=outcome2. In our example outcome1=head=outcome2. In real world, take for example a sample of realizations of a certain random var X with a pdf. Given the sample x1,...,xn of n observed realizations of X, and given the known pdf of the random variable, the Joint PDF calculated as the product above, will tell you what is the probability that ... cont $\endgroup$ – Fr1 Sep 8 at 15:33
  • $\begingroup$ ... the probability that if you observe n realizations of the random variable, then one realization is x1, another realization is x2,..., another is xn. So given a marginal pdf of the random variable, the joint pdf will allow you to calculate the probability that a set of realizations will have certain values denoted by x1,...,xn. Here I am using n realizations of a random variable observed N times. but I could have used N iid different random variables observed 1 times for each. So if you take two iid random variables A and B observed one time each, you are back to the previous example. $\endgroup$ – Fr1 Sep 8 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.