# How to write down the log-likelihood expression for this moving average model

Question is based on the paper Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models pdf download link

Assume the model is a FIR system of order 2 expressed as

$$y(n) = \sum_{m=0}^M \theta_mx(n-m) + v(n) \tag{1}$$.

The model can be rewritten as $$y(n) = \mathbf{\theta^T}x(n) + v(n) \tag{2}$$

The state space representation would be: $$y(n) = \theta_0 x(n) + \theta_1 x(n-1) + \theta_2 x(n-2) \tag{3}$$, and

$$y(n) = \theta^T x(n) + v(n) \tag{4}$$

Assuming the signal $x(n)$ is an $i.i.d.$ sequence of $(n \times 1)$ random vectors with independent components. $v(n)$ is an $i.i.d$ additive zero-mean Gaussian noise of unknown variance, $\sigma^2_v$. Here it still counts as a state space model, because $x(n)$ is still (albeit trivially) markov.

Question : Problem formulation : $\{x(n)\}$ (n by 1) is the unobserved input and $\{v(n)\}$ is the unobserved additive noise. The MLE solution will based on Expectation Maximization (EM). In the paper, I cannot understand how the likelihood expression would look like. Why do the Authors say mixture models?

What will be the complete likelihood and log-likelihood expression ?

This is how I proceeded.

I have used that fact that sum of 2 independent Gaussians is, again, Gaussian: $\theta^T x(n-M:n) + v(n) \sim \mathcal{N}(\theta^T \mu_x, \theta^T \Sigma_x \theta + 1)$.

Now, $\theta^T x(n-M:n)$ is Gaussian, too: a linear transformation of a Gaussian vector is still Gaussian. $\theta^T x(n-M:n) \sim \mathcal{N}(\theta^T \mu_x, \theta^T \Sigma_x \theta)$.

The variance of $y(n)$ is affected by $\theta$.

PDF of the observations $\mathbf{y}$ conditioned on the data sequence $x$ is

$$f(\mathbf{y|x,\theta}) = {\left(\frac{1}{\sqrt{\left(2\pi \sigma^2_v\right)}}\right)}^N \exp\left[-\frac{{\left(y-\theta^T x\right)}^2}{2\sigma^2_v}\right] \tag{1}$$ PDF of the complete data $\xi = [x, y]^T$:

$$f(\xi|\theta) = f(y|x,\theta)f(x) \tag{3}$$

• What is WGN? It looks as if your example has observable $\mathbf{H}$, but do you observe $u_n$? In a regular moving average model such as MA(1), $y_n=\varepsilon_n+\theta_1 \varepsilon_{n-1}$, the regressors are unobserved, therefore OLS does not work (at least directly). Maximum likelihood still does, though, and the maximum likelihood estimation of MA(1) and MA(q) processes is covered in a number of time series textbooks; Hamilton's "Time series analysis" should have that. Nov 30, 2015 at 20:12
• WGN = white Gaussian noise. I want to apply OLS technique to estimate $\theta$. Will the expression reduced to the same = $p\sigma^2_u/\sigma^2_v$ (Given in Steven Kay Book: Chapter 3: Estimation Theory of Signals Vol1) where $p$ is the MA order number and irrespective of the distribution of information signal $u$ is Bernoulli or not?In a supervised/non-blind setting how can I obtain the ML estimates off $\theta$?
– SKM
Dec 2, 2015 at 17:27
• Taking into account what I wrote in my previous comment, what is still unclear? I do not have the book you are citing, and I do not have the time to read it (so it would not help much even if I had the book). Of course, that does not say anything about other users, maybe someone will check it. Dec 2, 2015 at 19:10

"In the paper, I cannot understand how the likelihood expression would look like."

If they are using EM to estimate the model, then they are not using the (incomplete) likelihood function $f(y|\theta)$. They are using (evaluating) the complete likelihood function, which is the joint density of all $y$s and all $x$s. To get this, you would have to multiply your density $f(y|x, \theta)$ by $f(x|\theta)$. Edit: you have it on your last line in a general form, but you didn't write it out specifically for this model.

That's the beauty of EM. You can maximize something that depends on unobserved data using a clever iterative procedure. This is not what the contribution of the paper is, however.

"Why do the Authors say mixture models?"

It's because the observed data, which you denote $y(n)$ (different from the paper), is a mixture of the unobserved inputs, which you denote $x(n)$ (this is also different notation than the paper uses). If the dimension of $x(n)$ is one, however, this name might be slightly misleading. In either case, the observed data is a linear transformation of the unobserved data, with some extra noise added on top.

Edit: here is the complete data likelihood, as per request, with your notation (sort of).

\begin{align*} f(y_{1:T},x_{1:T};\theta) &= f(y_{1:T}|x_{1:T};\theta)f(x_{1:T};\theta) \\ &= \prod_{t=2}^T f(y_t|x_t) f(x_t|x_{t-1}) f(x_1) \\ &= \prod_{t=2}^T f(y_t|x_t) f(x_t) f(x_1) \\ &= \prod_{t=1}^T f(y_t|x_t) f(x_t) \end{align*}

with $f(y_t|x_t) = \text{Normal}(\theta ^T x_t, \Sigma_v)$

• Thank you for taking the time out to go through the paper. I am unable to write out the expression for the complete likelihood and it would really be helpful if you can mention it in your answer.
– SKM
Feb 3, 2016 at 3:53
• Okey dokey. I'll get to work on it in a bit. Feb 3, 2016 at 15:02
• Just a friendly reminder that you had said you would be helping me in getting the expression for the log-likelihood.
– SKM
Feb 8, 2016 at 21:25
• @SKM sorry for the delay. Is this what you're looking for? I couldn't find the distribution of each $x_t$, but you can fill that in yourself Feb 9, 2016 at 19:52
• The log of what I just added seems to be the third formula for section 2 Feb 9, 2016 at 19:55