# Beginner learning resources : Pdf and likelihood function for non-Gaussian time series model

I am struggling with exercise problems related to blind system identification where the knowledge about the source input is assumed to be known using maximum likelihood estimation of univariate time series processes which are non-Gaussian. For example, an AR model that is excited by non-Gaussian distribution like Bernoulli and then estimation from noisy time series where the process noise is Gaussian.

The AR(p) model be $y(t) = \phi_1y(t-1) + \phi_2y(t-2)+\ldots+\phi_py(t-p) + \eta(t)$

where $\eta$ is the stationary excitation input of white Bernoulli distribution and the process are stationary.

Let the output of the process be corrupted with measurement noise $v(t)$ which is white Gaussian noise of zero -mean and unknown variance $\sigma^2_V$: $z_{ar}(t) = y(t) + v(t)$

$v(t)$ is uncorrelated with $\eta(t)$.

Problem 1: What will be the joint pdf for the noiseless and the noisy model?

Problem 2: In the application of blind system identification, do we assume that the excitation input $\eta$ is zero mean with unknown variance?

Can somebody please provide references and guidelines to proceed to find the solutions? Thank you.

• Please write down a complete model for which you want to do this, with all the distributional assumptions. There are ambiguities that you must resolve. – Glen_b Dec 8 '14 at 5:37
• You might be best to stick to one problem in a question. When you say "$η$ is the stationary excitation input of white Bernoulli distribution", do you mean $\eta\stackrel{\text{iid}}{\sim}\text{Bernoulli}(p)$? – Glen_b Dec 8 '14 at 8:42
• @Glen_b: Yes, $\eta$ is $i.i.d$ and has Bernoulli pdf. I have put up one question. Thank you for your suggestions. – Ria George Dec 8 '14 at 15:55

Let $\Phi=(\phi_1,...,\phi_p)$ and $\mathbf{y}_{t-1} = (y_{t-1},...,y_{t-p})$

So your equations become $y_t = \Phi^{T}\mathbf{y}_{t-1} + \eta(t)$ and $z_{t} = y_{t} + v(t)$

thus $$z_t - \Phi^{T}\mathbf{y}_{t-1} = \eta(t) + v(t)$$ Where $p$ is the parameter of the Bernoulli distributed variable $\eta$; note that $$E[\eta(t) + v(t)]=p$$ and $$var[\eta(t) + v(t)]=p(1-p)+\sigma^2$$ Where $\sigma^2$ is the variance of $v(t)$.

Using this we construct the likelihood function $$\prod^{N}_{n=p+1} \mathcal{H}(z_t - \Phi^{T}\mathbf{y}_{t-1}\,|\,p\,,\,p(1-p)+\sigma^2)$$

where $\mathcal{H}$ is a pdf of unknown mean $p$ and variance $p(1-p)+\sigma^2$.

Personally I would assume $\mathcal{H}$ is a normal distribution since on average $\eta(t) + v(t) \sim \mathcal{N}(\,p\,,\,p(1-p)+\sigma^2)$

but this is fuzzy logic since it violates the assumption that $(\eta(t) + v(t))$ are identically distributed...but that's my 2 cents anyway.

• Thank you for your reply. Most of the points mentioned are unclear to me. Could you please clarify them?(1) In order to construct the pdf and the likelihood function for these cases, where the data points are from different distribution, do we always take pdf w.r.t to the noise terms? (2) Can you please mention how you calculated the variance? (3) Can you please mention the functional form of the expression for the likelihood function in terms of the 2 distribution? (4) How do you say that assuming $H$ to be normal will violate the i.i.d assumption? – Ria George Dec 15 '14 at 14:19