I'm estimating the coefficients of variation (CV) for features in a dataset. Each of the features is well represented by a normal distribution. I've done the estimate in two ways, both using bayesian methods. In each case I fit the model to zscores then invert the transform for the samples. The two approaches are

  1. independently estimating the mean $\mu$ and standard deviation $\sigma$ for each feature, with standard normal priors for the means and exponential priors (scale=1) for the standard deviations
  2. fitting a multivariate normal distribution to the whole dataset, using the LKJCholeskyCov prior from PyMC with the same exponential prior distributions for the diagonal of the covariance matrix and eta=2.

The results are slightly different in these two cases. The only difference is the multivariate distribution incorporates covariance parameters, because the priors for the variance of each parameter are the same in each model.

What explains the difference? Why does including covariance parameters change the estimates for mean and variance?

  • $\begingroup$ Please tell us specifically how you are estimating the parameters. You hint at a Bayesian approach, where the estimate depends on the prior you choose. The standard methods (MoM or MLE) do yield exactly the same estimates in both cases. $\endgroup$
    – whuber
    Commented Jul 24, 2023 at 19:22
  • $\begingroup$ I added some more context about the priors $\endgroup$
    – qsfzy
    Commented Jul 24, 2023 at 19:33
  • $\begingroup$ In your first case, you are, in effect, forcing all the off-diagonal elements of the estimated covariance matrix to equal zero. In the second case, you are not. $\endgroup$
    – jbowman
    Commented Jul 24, 2023 at 20:12
  • $\begingroup$ I am aware of the structural difference between the two models. I'm asking why simultaneous estimation of the off diagonals should change the diagonals. $\endgroup$
    – qsfzy
    Commented Jul 24, 2023 at 21:19
  • 1
    $\begingroup$ I would suggest the question would be more fruitfully asked in the other direction: how could imposing any prior on the off-diagonal elements not affect the estimate? For intuition, consider an extreme example of such a prior that assigns high probability to perfect correlation in the case of $2$ features. That would effectively link any estimates of the moments of those features and thereby ought to affect them both. $\endgroup$
    – whuber
    Commented Jul 26, 2023 at 20:32


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.