0
$\begingroup$

In my previous question Density function for AR model, the density function of AR model has the covariance-variance matrix given as $\sigma^2 *V_p$. In multivariate Gaussian distribution, the pdf contains no $\sigma^2$ term.

I have conceptual questions based on this.

(1) Does AR or ARMA or MA model of order greater than 1 when excited by white Gaussian noise, qualify as multivariate Gaussian distributution?

(2) Why the scalar variance is multiplied with the covariance and why do we need the covariance matrix? Why are these 2 expressions (in the links) of pdf different? In Tutorial1 Sec 3, the joint pdf for AR(p) model contains the variance of white noise multiplied with the Covariance Matrix. But, In Tutorial 2 Page#9 THE ARMA model is presented, there is only the covariance matrix in the expression !! This is confusing. For AR(1) we only consider the scalar $\sigma^2$ but for AR(p) why do we need covariance matrix as there is only 1 random variable and the correlation or covariance for 1 random variable does not make sense. Please correct me if wrong. Will appreciate intuitive and detailed answer. Thank you

Improve this question
$\endgroup$
4
  • $\begingroup$ Is the $\sigma^2$ you are talking about in the AR model the variance of the white noise? The variance of a vector of length n is necessarily an n by n variance-covariance matrix, not a scalar. That may depend on the white noise variance, and so $\sigma^2$ may even show up as a factor, but doesn't make it the "scalar variance of the vector". Perhaps this is the underlying confusion? $\endgroup$
    – Silverfish
    Commented Nov 29, 2014 at 0:06
  • $\begingroup$ @Silverfish: I am talking about varaince of white noise in the AR model (not a vector AR model, a simple AR of 1 variable & order> 1; or ARMA model). When order=1, there is no matrix. Otherwise, there is. Why? Since, there is only 1 random variable in AR model, I thought it is not a vector of random variable. So, why is the distribution called "Multivariate"?I thought variance is a scalar since during implementation we create the white noise as a random vector and multiply it with the square root of variance. Thus, the confusion as to how the expression gets a matrix of "variance-covariance"? $\endgroup$
    – Ria George
    Commented Nov 29, 2014 at 20:07
  • $\begingroup$ The response variable is indexed over time. Are you sure you're not looking at the joint pdf of the vector of observations $(Y_1, Y_2, ..., Y_n)$? $\endgroup$
    – Silverfish
    Commented Nov 29, 2014 at 20:37
  • $\begingroup$ Yes, the pdf is joint pdf but there is only 1 random variable Y. I came across this Question stats.stackexchange.com/questions/125642/… where the response is that for AR model the correlation E{Y*Y'} does not make sense as there is only 1 random variable ( justification being that Correlation is calculated for vector of r.v). If so, then how does covariance can be calculated as cov is also between different random variables. I have so many fundamental doubts & Questions that cannot be asked to the course instructor :( $\endgroup$
    – Ria George
    Commented Nov 29, 2014 at 20:42

2 Answers 2

Reset to default
2
$\begingroup$

The covariance matrix is essential to characterize the joint distribution of random variables. Hence, it is not sufficient to only use one variance parameter. In the case of a multivariate Normal distrubtion, you have some normally distributed random variables that are also jointly normally distributed. For example, take two normally distributed random variables $X1$ and $X2$ with $X1 \sim N(3,1) $ and $X2 \sim N(2,4)$. Assume, they are independent. This implies the covariance is zero. Now, one can show the joint distribution is also Normal with $N \left( \left( \begin{array}{ccc} 3 \\ 2 \end{array} \right), \left( \begin{array}{ccc} 1 & 0 \\ 0 & 4 \end{array} \right) \right) $.

In the picture below, you can see a sample and a contour plot taken from the stated joint distribution. There is now dependence but look at the scales of the axis and how the spread of the points is determined by the variance of each variable. If they had the same variance, there would be no difference.

enter image description here

Now, assume the $X1$ and $X2$ are still also jointly normally distributed but not independent and exhibit a covariance of $1.5$. This means, there is some linear dependence between them and the distribution is now $N \left( \left( \begin{array}{ccc} 3 \\ 2 \end{array} \right), \left( \begin{array}{ccc} 1 & 1.5 \\ 1.5 & 4 \end{array} \right) \right) $. The picture below shows a dramatically changed distribution. Now, high values of $X1$ correspond tho high values of $X2$ and vice versa. The important point is, whenever multivariate analysis is done, one tries to find the dependence between random variables and in the real world one can find a many dependent variables. So, that makes it clear why a covariance matrix is needed to characterize a joint distribution and why it is important.

enter image description here

Regarding your first question: Your given matrix is in the AR model multiplied by a scaler because for the purpose of simplicity homoscedasticity is often assumed. So this just means the variance does not depend on time in this model.

Note: For the illustration, I used the codes of the socalled Quantnet provided by the chair of statistics at the Humboldt University of Berlin. Click here and you can download the code and play with the structure of the covariance matrix to see how it behaves in this two dimensional example.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Thank you for your illustrations & effort. I have conceptual issues in understanding the basics. Could you please clarify?(1) When a process contains 1 varaible, I think there is only 1 random variable & not a vector of random variable. Am I correct? (2) When the order of such a process is 1, then we consider the variance of white noise in the AR model or ARMA model to be a scalar. Otherwise, when model order >1 there is a matrix. The process is stationary & correlated samples - prev samples affect the future), so why is there a matrix of co-variance ? $\endgroup$
    – Ria George
    Commented Nov 29, 2014 at 20:13
  • $\begingroup$ (3) If there is only 1 random variable in a model of any order, I thought it is not a vector of random variable. So, why is the distribution called "Multivariate"? This term seems fit for ARMA when we have random variables for AR, & MA.(4) I thought variance is a scalar for AR model of any order since during implementation we create the white noise as a random vector and multiply it with the square root of variance. Thus, the confusion as to how the expression gets a matrix of "variance-covariance"? $\endgroup$
    – Ria George
    Commented Nov 29, 2014 at 20:19
  • $\begingroup$ I am not a time series expert but I think if you have an AR(1) process and want to find the joint distribution of X(T) and X(T-1), you also need the covariance. This is also important to find the parameter (see Yule Walker equations). The variance in a AR(1) is a scaler, because it is assumed not to depend on t. Also, if the model is of higher order, say p>1. You need this matrix. It is, however, not a usual covariance matrix but a auto-covariance matrix. This is necessary, because you want to know how the joint distribution behaves and infer some parameters from it. $\endgroup$
    – random_guy
    Commented Nov 29, 2014 at 20:51
  • 1
    $\begingroup$ Don't let it confuse you. It's mostly just notation and notation differs even in the same topic. Multiplying a matrix with a scalar variance only indicates all entries in the covariance matrix are proportional to this variance. In the expression $\sigma^2 V$, you can see $V$ as matrix with ones on the diagonal and the correlation coefficients as off diagonal elements. Since $\sigma$ is assumed to be equal for all $X(T)$. The correlation between two elements is $\rho(X_T,X_{T-s}) = Cov(X_T,X_{T-s})/\sigma$. This implies the covariance is $Cov(X_T,X_{T-s}) = \rho(X_T,X_{T-s})/\sigma$. $\endgroup$
    – random_guy
    Commented Nov 30, 2014 at 10:57
  • 1
    $\begingroup$ Thank you for your intuitive answer. I think there is a typo in your last expression. Should it be $Cov = \sigma^2 \rho$. Also, which cases do we assume/consider $\sigma^2*V$ and only $\sigma^2$? Is it possible that you may update your answer with this intuitive part? Thank you for your efforts. $\endgroup$
    – Ria George
    Commented Nov 30, 2014 at 20:12
2
$\begingroup$

In general, the covariance matrix of a multivariate normal distribution is denoted $\Sigma$. $\Sigma$ contains variances and covariances. When $\Sigma$ is parametrised (see, e.g, Table 56.13 and 56.14), it is useful to rewrite $\Sigma$ in order to reveal its structure and to see the role of its parameters. Below is a basic example.

Let $Y_1, \ldots, Y_n$ have a multivariate distribution with covariance matrix $\Sigma$. Further, suppose that

  • $\text{Var}(Y_i) = \sigma^2$ for all $i$;
  • $\text{Cov}(Y_i, Y_j) = 0$ for all $i$ and $j$ ($i \neq j$).

(That is, $\Sigma$ is parametrised in terms of a single parameter, $\sigma^2$, the common variance.)

Then,

$\Sigma = \left( \begin{array}{cccc} \sigma^2 & 0 & \ldots & 0 \\ 0 & \sigma^2 & \ldots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \ldots & \sigma^2 \end{array} \right) = \sigma^2 \left( \begin{array}{cccc} 1 & 0 & \ldots & 0 \\ 0 & 1 & \ldots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \ldots & 1 \end{array} \right) = \sigma^2 I_n$

Another example

Let $Y_1$ and $Y_2$ have the same variance $\sigma^2$. The correlation matrix can be written as $$ R=\left( \begin{array}{cc} 1 & \frac{\text{Cov}(Y_1, Y_2)}{\sigma \times \sigma} \\ \frac{\text{Cov}(Y_1, Y_2)}{\sigma \times \sigma} & 1 \end{array} \right) $$ Then the covariance matrix is $\sigma^2 R$.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your reply. This is quite different from the expression for AR(p) model where it is $sigma^2 \Sigma$. I have not come across this relation before and mostly familiar with the one that you have mentioned. That is why I raised the Question. Could you also explain for Ar(p) model in the similar way why we are multiplying $\sigma^2$ with $\Sigma$? $\endgroup$
    – Ria George
    Commented Nov 28, 2014 at 18:39
  • $\begingroup$ Thank you for your updated reply. I am confused due to the terminologies - The $\sigma^2$ is the variance of white noise in this case. (1) When a process contains 1 variable, there is only 1 random variable & not a vector of random variable. (2) When the order of such a process is 1, then we consider the variance of white noise in the AR model or ARMA model to be a scalar. Otherwise, when model order >1 there is a matrix. The process is stationary & correlated samples - prev samples affect the future), so why is there a matrix of co-variance ? $\endgroup$
    – Ria George
    Commented Nov 29, 2014 at 20:32
  • $\begingroup$ Moreover, during implementation the noise vector is created by multiplying square root of variance with a random vector. So, where from are we getting a multivariate distribution. When the AR model order >1, the distribution is called multivariate & hence is the variance covariance called a matrix. But, the number of random variables remains the same as in AR process - 1 random variable due to the AR model & another for the noise part. I understand that for ARMA the situation is different. In These may sound too trivial but such Questions cannot be asked to the instructor, alas! $\endgroup$
    – Ria George
    Commented Nov 29, 2014 at 21:00
  • $\begingroup$ econ.nsysu.edu.tw/ezfiles/124/1124/img/… in this link, the joint pdf of AR(1) does not contain the covariance matrix. But for higher order, there is the covariance and variance matrix, where variance is a scalar multiplied with the covariance matrix. So, when to consider a matrix and when not? $\endgroup$
    – Ria George
    Commented Nov 29, 2014 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.