3
$\begingroup$

I need to find the pdf of x, $f_x(x)$ which is the sum of two random variables $u$ and $w$ and they are independent. I have found the pdf but I am unsure if it is correct or not, the expression is quite complicated. I will really appreciate if somebody takes the time out to let me know if it is correct and rectify otherwise. The model is $x = u + v$,

The pdf are :

$f(u) = (1-\epsilon) \phi_b(u) + \epsilon \phi_l(u), 0<\epsilon<1 $ and

$f(v) = \frac{1 }{\sqrt{2 \pi \sigma_v^2}}\exp(\frac{-v^2(n)}{2\sigma^2_v}) $

and

$ \phi_b(u) = \frac{1 }{\sqrt{2 \pi \sigma_b^2}}\exp(\frac{-u^2(n)}{2\sigma^2_b})$ ;

$ \phi_l(u) = \frac{1 }{\sqrt{2 \pi \sigma_l^2}}\exp(\frac{-u^2(n)}{2\sigma^2_l})$ ;

Steps:

$f(x) = \int_{0}^\infty f(x-u) f(u) du $

$ = \int_{0}^\infty \frac{1 }{\sqrt{2 \pi \sigma_v^2}}\exp(\frac{-(x-u)^2}{2\sigma^2_v}) \bigg[(1- \epsilon) \phi_b(u) + \epsilon \phi_l(u)\bigg] du $

$ = \int_{0}^\infty \frac{1 }{\sqrt{2 \pi \sigma_v^2}}\exp(\frac{-(x-u)^2}{2\sigma^2_v}) \bigg[\frac{(1-\epsilon) }{\sqrt{2 \pi \sigma_b^2}}\exp(\frac{-u^2(n)}{2\sigma^2_b}) + \frac{\epsilon }{\sqrt{2 \pi \sigma_l^2}}\exp(\frac{-u^2(n)}{2\sigma^2_l}) \bigg] du$

$ = \int_{0}^\infty \frac{1 }{\sqrt{2 \pi \sigma_v^2}}\exp(\frac{-(x-u)^2}{2\sigma^2_v}) \bigg[\frac{(1-\epsilon) }{\sqrt{2 \pi \sigma_b^2}}\exp(\frac{-u^2(n)}{2\sigma^2_b}) \bigg] du + \int_{0}^\infty \frac{1 }{\sqrt{2 \pi \sigma_v^2}}\exp(\frac{-(x-u)^2}{2\sigma^2_v}) \bigg[\frac{\epsilon }{\sqrt{2 \pi \sigma_l^2}}\exp(\frac{-u^2(n)}{2\sigma^2_l}) \bigg]$

$ = \frac{(1-\epsilon) }{\sqrt{2 \pi \sigma_v^2} * \sqrt{2 \pi \sigma_b^2}} \int_{0}^\infty \exp(\frac{-(x-u)^2}{2\sigma^2_v})\exp(\frac{-u^2(n)}{2\sigma^2_b}) du + \frac{\epsilon }{\sqrt{2 \pi \sigma_v^2}* \sqrt{2 \pi \sigma_l^2} } \int_{0}^\infty \exp(\frac{-(x-u)^2}{2\sigma^2_v}) \exp(\frac{-u^2(n)}{2\sigma^2_l}) $

$ = \int_{0}^\infty A(u) du + \int_{0}^\infty B(u) du $

Now,

$ \int_{0}^\infty \exp(\frac{-(x-u)^2}{2\sigma^2_v})\exp(\frac{-u^2(n)}{2\sigma^2_b}) du = \frac{ \sigma_v \sigma_b \sqrt{\pi/2} \exp\big(-x^2/2(\sigma^2_v + \sigma^2_b)\big) erf\big(\sigma^2_v u + \sigma^2_b(u-x)/( \sigma_v \sigma_b \sqrt{2} \sqrt{\sigma^2_v + \sigma^2_b})\big)}{\sqrt{\sigma^2_v + \sigma^2_b}} $ and

$ \int_{0}^\infty \exp(\frac{-(x-u)^2}{2\sigma^2_v}) \exp(\frac{-u^2(n)}{2\sigma^2_l}) = \frac{ \sigma_v \sigma_b \sqrt{\pi/2} \exp\big(-x^2/2(\sigma^2_v + \sigma^2_l)\big) erf(\sigma^2_v u + \sigma^2_l(u-x)/ \sigma_v \sigma_l \sqrt{2} \sqrt{\sigma^2_v + \sigma^2_l})}{\sqrt{\sigma^2_v + \sigma^2_l}} $

After manipulation, the final pdf is coming out to be : $f(x) = \frac{\bigg ((1- \epsilon) \exp \big(\frac{-x^2}{2(\sigma^2_v + \sigma^2_b)} \big)\big[1 - erf\big( \frac{- x\sigma^2_b}{\sigma_v \sigma_b \sqrt{2(\sigma^2_w + \sigma^2_b)}} \big) \big] \bigg)}{2 \sqrt{2 \pi(\sigma^2_v + \sigma^2_b)}} + \frac{\bigg (\epsilon \exp \big(\frac{-x^2}{2(\sigma^2_v + \sigma^2_l)} \big)\big[1 - erf\big( \frac{- x\sigma^2_l}{\sigma_v \sigma_l \sqrt{2(\sigma^2_w + \sigma^2_l)}} \big) \big] \bigg)}{2 \sqrt{2 \pi(\sigma^2_v + \sigma^2_l)}} $

I am having serious doubt of committing mistake since ultimately I need to take the log likelihood of this complicated expression. Please help.

$\endgroup$
  • 1
    $\begingroup$ This is confusing because you start by defining "$f$" twice in different ways and then you use the same symbol in a third way to represent a convolution! I would guess the first one is a mixture of normals and the second is a normal distribution, and that you want to compute their convolution. But since you use the same symbol for each, you haven't a prayer of getting the right answer. The answer can easily be found because the convolution of two normals is also normal; no integration at all is needed. $\endgroup$ – whuber Jan 21 '15 at 23:58
  • 2
    $\begingroup$ What is $n$ in $f(v) = \frac{1 }{\sqrt{2 \pi \sigma_v^2}}\exp(\frac{-v^2(n)}{2\sigma^2_v})$? This problem can be done far more easily by recognizing that the sum of independent normal random variables is a normal random variable, and that the pdf of $U$ is a mixture of normal densities. Thus, $$f_U\star f_V = ((1-\epsilon)f_A+\epsilon f_B)\star f_V = (1-\epsilon)f_A\star f_V +\epsilon f_B\star f_V$$ and computing the two convolutions separately. $\endgroup$ – Dilip Sarwate Jan 22 '15 at 0:01
  • 1
    $\begingroup$ Srishti: When you say "sum of a mixture distribution with another distribution" please note that you don't mean what you said. What you mean is something like "distribution of a sum of two random variables, one of which has a mixture distribution". The actual sum of two distributions ($F+G$, say) is not itself a distribution. This is a very common error, so I can hardly blame people for repeating it (no doubt I have fallen into this mistake myself on occasion). $\endgroup$ – Glen_b Jan 22 '15 at 1:20
  • $\begingroup$ @DilipSarwate: $n$ denotes the index of the sample $v$. I am new to convolution, so I am unfamiliar with the details. I got the same terms when I did the convolution as yours, so should I not integrate $f_u*f_v$ to get the result? Integration complicated the matter but from Whubers answer and from the comments, it appears that I don't have to do integration. So, how does one compute the convolution without integration? $\endgroup$ – Srishti M Jan 22 '15 at 2:00
7
$\begingroup$

Let's start with notation. When you need to refer to different things, you have to give them different names. To that end let $f$ refer to the first PDF and $g$ the second.

Second, let's focus on the form of the PDFs and strip away unnecessary details:

  • $f$ is a positive linear combination of $\phi_b$ and $\phi_l$: that is called a mixture.

  • $\phi_b$ and $\phi_l$ are both of the form

    $$\phi(t) = \text{some number}\times \exp(-t^2/2\sigma^2)$$

    where $\sigma$ is a parameter. These are recognizable as describing Normal distributions with mean $0$ and standard deviations $\sigma_b$ and $\sigma_l$. (I am ignoring the "$(n)$" that appears in the original expressions, because this makes no sense.)

  • $g$ is likewise recognizable as a Normal distribution with mean $0$ and standard deviation $\sigma_v$.

It is elementary that the sum of a Normal$(\mu, \sigma^2)$ distribution (having mean $\mu$ and standard deviation $\sigma$) and a Normal$(\mu^\prime, \sigma^{\prime 2})$ distribution is a Normal$(\mu+\mu^\prime, \sigma^2 + \sigma^{\prime 2})$ distribution. It is also immediate from the mathematical definitions that convolution (which is the operation used to carry out the sum of distributions) is bilinear: in other words, when you convolve some distribution with a mixture, you get the mixture of the convolutions with the components.

Therefore, without any more consideration, we may conclude that the distribution of the sum is the mixture of a Normal$(0, \sigma_v^2 + \sigma_b^2)$ distribution (with coefficient $1-\epsilon$) and a Normal$(0, \sigma_v^2 + \sigma_l^2)$ distribution (with coefficient $\epsilon$). If you really, really want to see its PDF in all its glory, here it is:

$$\newcommand{\s}[2]{\sigma_{#1}^2 + \sigma_{#2}^2} \newcommand{\N}[2]{\frac{#1}{\sqrt{2\pi(#2)}}\exp\left(\frac{-t^2}{2(#2)}\right)} \eqalign{&(f*g)(t) = \\ &\N{1-\epsilon}{\s{v}{b}} + \N{\epsilon}{\s{v}{l}}.}$$

Normally this level of detail is not needed--it just obscures the basic simplicity of what is going on.


Appendix

This approach can rigorously be justified by considering characteristic functions (cf). The cf of any random variable $X$ with distribution $F$ is a function of a real variable $t$ defined by

$$\newcommand{\E}[2][F]{\mathbb{E}_{#1}{\left(#2\right)}} \phi_F(t) = \E{e^{itX}} = \E{\cos(tX)} + i\E{\sin(tX)}$$

(where, as usual, $i^2 = -1$). Because $|e^{itX}|\le 1$, $|\phi_F(t)|\le \E{1}=1$ exists and is finite for all $t$.

The elementary properties of expectations and exponentials provide simple useful algebraic relationships between characteristic functions, sums, and mixtures:

  1. Let $X$ and $Y$ be independent random variables with distributions $F$ and $G$, respectively, and let $\alpha$ and $\beta$ be numbers. Then

    $$\phi_{\alpha X + \beta Y}(t) = \E[(F,G)]{e^{i(\alpha X + \beta Y)t}} = \E[(F,G)]{e^{i X (\alpha t)} e^{i Y (\beta t)}} = \phi_F(\alpha t)\phi_G(\beta t).$$

  2. Let $X$ be governed by the distribution $(1-\epsilon)F + \epsilon G,\ 0 \le \epsilon \le 1$ be a mixture of the distributions $F$ and $G$. (Let us call this an "$\epsilon$ mixture of $X$ and $Y$".) Because expectations are integrals and integrals are linear in their arguments,

    $$\eqalign{ \phi_{(1-\epsilon)F + \epsilon G}(t) &= \E[(1-\epsilon)F + \epsilon G]{e^{i t X}} \\ &= (1-\epsilon)\E[F]{e^{i t X}} + \epsilon\E[G]{e^{i t X}} \\ &= (1-\epsilon)\phi_F(t) + \epsilon \phi_G(t). }$$

  3. A fundamental theorem asserts that the characteristic function completely determines the distribution. This comes down to showing how $F$ can be recovered from $\phi_F$, which involves more detailed analysis of the integrals involved. An account can be found in textbooks on probability, real analysis, or Fourier analysis.

One actual calculation of an integral is needed. By definition, the standard Normal distribution has the characteristic function $\phi(t) = e^{-t^2/2}$. This corresponds to the distribution $F$ whose density function is proportional to $dF(x) = e^{-x^2/2}$ (with constant of proportionality $C$, say: its actual value does not matter to us). This can be verified by computing

$$\eqalign{ \phi_F(t) &= \E{e^{i t X}} = C\int_{\mathbb R} e^{i t x} e^{-x^2/2} dx \\& = C\int_{\mathbb R} e^{-[(x - i t)^2 + t^2]/2} dx \\& = e^{-t^2/2}C\int_{\mathbb R} e^{-(x - i t)^2 /2} dx \\& = e^{-t^2/2}C\int_{\mathbb{R}-it} e^{-x^2 /2} dx \\& = e^{-t^2/2}. }$$

The only calculation of an integral occurred at the last step; everything else was elementary algebraic manipulation. The justification relies on Cauchy's Integral Theorem. if you know this theorem and its applications to integration in the Complex plane, then you will also recognize that this calculation required only establishing that $e^{-z^2}$ grows very small when the real part of $z$ is large and the imaginary part is bounded.

The "Normal distribution with mean $\mu$ and variance $\sigma^2$" is defined to be the distribution of $\sigma X + \mu$ for a standard Normally distributed random variable $X$: this is a linear combination of a standard Normal and a constant, with coefficients $\sigma$ and $\mu$ respectively. Consider two such Normal distributions, both with zero mean and variances $\sigma$ and $\sigma^{\prime 2}$. By (1) their cf is

$$\phi(t) = \phi_F(\sigma t)\phi_F(\sigma^\prime t) = e^{-(\sigma t)^2/2}e^{-(\sigma^\prime t)^2/2} = e^{-(\sigma^2 + \sigma^{\prime 2}) t^2/2}.$$

The latter is recognizable as the cf of a Normal$(0, \sigma^2 + \sigma^{\prime 2})$ distribution. Because (by (3) above) the cf determines the distribution, we conclude the sum of independent Normal distributions is again Normal. Its variance is the sum of the variances of the components.

Here, then, is the fully rigorous solution to the problem. First, its statement:

Let $X, Y,$ and $Z$ be independent random variables with Normal$(0,\sigma_b^2)$, Normal$(0,\sigma_l^2)$, and Normal$(0,\sigma_v^2)$ distributions, respectively. What is the sum of $Z$ with an $\epsilon$ mixture of $X$ and $Y$?

Relationships (1) and (2) above, along with the result about sums of independent Normals give the answer so immediately and clearly that all one has to do is write it down. Even if you want to display the little details, all the calculations are simple algebra because no more integration is necessary: it is all hidden in the one integral we did to compute the standard Normal cf from its pdf.


In a generalization of this problem it is required to find the distribution of a sum of two independent variables where one is a mixture of two independent variables. Let the three distributions involved be $F$, $G$, and $H$. The basic calculation for the cf of this sum is still the same:

$$\phi(t) = (1-\epsilon)\phi_F(t)\phi_H(t) + \epsilon\phi_G(t)\phi_H(t).$$

It is manifestly another mixture. To obtain formulas for the pdf of this distribution, then, you have to find pdfs for which $\phi_F(t)\phi_H(t)$ and $\phi_G(t)\phi_H(t)$ are their cfs. Sometimes this can be done by inspection, as we did previously; sometimes it can be done using formulas to compute pdfs from cf using inverse Fourier transforms; and sometimes it just cannot be done in any nice closed analytic form at all.

$\endgroup$
  • $\begingroup$ Thank you for explaining clearly & sorry for the confusing notations. Just to confirm if I have followed, (a) I got the form of the pdf $f(u)$ from the paper "Efficient estimation of parameters for non-Gaussian autoregressive processes" by Debasis Sengupta & Steven Kay. The Authors have mentioned that the pdf for a non-Gaussian process can be modeled by a Gaussian mixture. So, is the final pdf for the random variable $x$ also termed as Gaussian mixture or bivariate Gaussian mixture because it has come from 2 r.v? $\endgroup$ – Srishti M Jan 22 '15 at 1:54
  • $\begingroup$ (b) the pdf of $x$ which is the sum of 2 r.v $u + w$ as given by your answer $(f*g)(x)$ where I will replace $t$ with $x$ as I want the pdf of the random variable $x$. Then, why is my integration procedure incorrect and I did not get the same result? Is it because for trivial case such as this, integration is not allowed? Why did you not do integration since I know that in order to compute pdf from convolution, integration is required. Could you please explain this a bit more? Thank you very much. $\endgroup$ – Srishti M Jan 22 '15 at 2:01
  • $\begingroup$ In order to compute the pdf by convolution, we need to express the unknown in terms of the known. SO, why this will not work here as I mentioned in my Question? The reason I ask is how would I go about when I need to find the distribution from the sum of a mixture model with another non-Gaussian distribtuion say Binomial or Poisson. $\endgroup$ – Srishti M Jan 22 '15 at 2:09
  • 2
    $\begingroup$ I added some material to address these comments. $\endgroup$ – whuber Jan 22 '15 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.