7
$\begingroup$

Consider multivariate time series models that estimate potentially time-varying conditional means, variances, and correlations (one type of model might be a VAR(p)+Garch(1,1)+DCC Gaussian Copula model). I will simulate from one "true" distribution and compare other potential models to it. I don't particularly care about RMSE because I care just as much about the conditional covariance matrix as the conditional means. For the sake of simplicity, assume that the conditional mean and conditional covariance are sufficient to explain "true" distribution.

I can calculate the conditional mean vector and conditional covariance matrix of each model and the true model at any given point in time. Alternately, I can calculate the forecasted conditional means and covariances on a rolling basis.

  1. Is there any reason to prefer the Kullback-Leibler divergence or Bhattacharyya distance as comparison methodologies or should I just do both? Alternately, is there a different method that is even more appropriate.
  2. In the constant multivariate Gaussian case, is it possible to make statistical inferences based on these comparison methodologies, such as that the divergence is not statistically bigger than zero or that the divergence of one is bigger than another?
  3. If I calculate the conditional means and variances at any given point in time, I could calculate the comparison statistic in each period. This would produce a time series of divergences/distances. Is it reasonable to make statistical inferences in this case by just assuming it is a typical time series?
Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ I don't get why you are throwing away essentially all the reputation you have accumulated so far to hope for an answer to this. The reason it is not getting attention is probably because it is so specicalized that no one who has seen it thus far has the expertise to answer it. If you get no answer for the next six days you lose 50 reputation points and get no benefit. $\endgroup$
    – Michael R. Chernick
    Commented Jun 1, 2012 at 19:26
  • $\begingroup$ I got +100 reputation for joining since I have more reputation on another stackexchange site. Since the question is important, figured it couldn't hurt. $\endgroup$
    – John
    Commented Jun 4, 2012 at 16:23
  • $\begingroup$ Why would you compare means and covariances between models, where one is such that you know it's true? Or are you trying to say you're knowingly fitting 'wrong' models on (real or simulated) data, and trying to come up with a routine that flags when something is wrong or when the model is dangerous or unsuitable or fits poorly? Is this for a trading strategy? $\endgroup$
    – Taylor
    Commented Nov 2, 2016 at 3:43
  • $\begingroup$ @Taylor The idea was that I cared about the density as much as the mean. This question was from a few years ago. Nowadays I would probably use WAIC or Leave one out cross validation. $\endgroup$
    – John
    Commented Nov 2, 2016 at 12:40
  • $\begingroup$ @John my fault I didn't see the datestamp $\endgroup$
    – Taylor
    Commented Nov 2, 2016 at 15:32

1 Answer 1

Reset to default
2
$\begingroup$

There are some methods out there for two sample testing of covariance matrices but no one has specifically looked at testing for conditional covariance matrices. Are you interested in an overall differences between covariances or differences in specific rows of the covariance matrices or recovering the exact support of the difference ? If the first, you might be able to use existing literature on two sample testing of covariance matrices. This would definitely work for point 2 in your question. This is really an open area of research and quite under explored in the time-varying setting. An alternative to KL-distances is using the maximum t-statistic across all the entries of your conditional covariance matrix.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the reply. I wasn't aware of how "open" an area of research this was. To your point, I care about BOTH the conditional mean vector and the conditional covariance matrices. The benefit of the two metrics I mention is that they allow for comparisons of both the mean and covariance. Other statistics, like AIC/BIC, are problematic since I don't plan on performing ML jointly for everything. $\endgroup$
    – John
    Commented Jun 4, 2012 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.