2
$\begingroup$

Let ${M_1, M_2}$ denote two competing forecasting models.

With Bayesian model averaging we can get

$p(y_{T+h}|y_{1:T}) = \sum_{j=1}^2p(y_{T+h}|y_{1:T},M_j)*p(M_j|y_{1:T})$

$1:T$ represents the training set and $h$ the h-ahead forecast of a out-of-sample set $N$

My problem is now to compute the j-th posterior model probalitites (PMP):

$p(M_j|y_{1:T}) = \frac{p(y_{1:T}|M_j)*p(M_j)}{\sum_{l=1}^2p(y_{1:T}|M_l)*p(M_l)}$

Assuming equal prior weights the function reduces to

$p(M_j|y_{1:T}) = \frac{p(y_{1:T}|M_j)}{\sum_{l=1}^2p(y_{1:T}|M_l)}$

My problem is now, that I dont know how to compute $p(y_{1:T}|M_j)$.

I have the densities/histograms of the training-data (realized data) as well as from both models for the training-data.

Is this enough to compute the above marginal likelihood for each model? Are there any useful approximations?

Can I use maybe use BIC for the weights?

Improve this question
$\endgroup$
2
$\begingroup$

I would frame the problem this way:

$p(M_j|y_{1:T}) = \frac{\big(\int{p(y_{1:T}|\theta,M_j)p(\theta|M_j)d\theta}\big)p(M_j)}{\sum_{l=1}^2\big(\int{p(y_{1:T}|\theta,M_l)p(\theta|M_l)d\theta} \big) p(M_l)}$

where $\theta=(\theta_1,\dots,\theta_d)$ is the vector of random parameters on which your models depend. Actually, $d$ may be different for different models, but since you have only two models, I'll keep the notation simple. If you have $d_1\neq d_2$, then in the following just consider $d=\max{(d_1,d_2)}$. From the above formula, it's obvious that you need two "ingredients":

  1. you need prior probabilities for the models. You assumed that $p(M_1)=p(M_2)=0.5$, so that's already taken care of, but it's good to remember that your results are conditional on this assumption.
  2. you need the marginal likelihoods for each model, or equivalently the Bayes factors. That is to say, for each model $M_j$ you need to integrate the likelihood $p(y_{1:T}|\theta,M_j)$ with respect to $p(\theta|M_j)$, which is the prior distribution of $\theta$ for model $M_j$. Now, in a discrete case, where $\theta$ can assume only one possible value under model $M_j$, this step is easy. For example, this is often the case for the model derived under the null, in Bayesian hypothesis testing. But in the continuous case, this becomes a bit more complicated. If $p(y_{1:T}|\theta,M_j)$ and $p(\theta|M_j)$ are a conjugate pair, then this is easy, thus I assume they aren't. I can think of 3 options:
    • if $T\gg d$, then the likelihood will be extremely peaked around the MLE of $\theta$. In this case, the Laplace approximation should work well. Another large sample approximation is the one based on the BIC. However, I don't know a lot about it: I only used it for linear regression models. Since you're talking about forecasts, I guess you're doing time series modeling. I know that BIC is used also for time series modeling. I would at least cross-check with another method.
    • since you have an explicit expression for $p(\theta|M_j)$, you can sample from it. This means that you can compute $\int{p(y_{1:T}|\theta,M_j)p(\theta|M_j)d\theta}$ by Monte Carlo integration ( unless of course you chose a Cauchy prior). Since likelihood functions are nasty beasts, usually this will converge slowly, because most of the MC samples $\theta_i$ will correspond to very small likelihood values. Importance sampling will improve upon that.
    • if $d$ is small (say, less than 9) numerical integration may work very well. Sparse grid Gaussian quadrature or adaptive Gaussian quadrature will do the trick. If $d$ is very small (for example, less than 4 or 5), even tensor grid Gaussian quadrature could work.
Improve this answer
$\endgroup$
  • $\begingroup$ thanks. I am not sure what $θ=(θ1,…,θd)$ should represent in my case. The one of the two models is using historical data (variances and covariance of data of 2 variables(stock return and market index return) to forecast a new observation, while the other one is using externally derived values and just adjust it a bit... There is no time-series-modelling actually. The models are fixed and the parameters should not be varied. $\endgroup$ – Plazi Nov 3 '16 at 19:32
  • $\begingroup$ In the first model, it looks like the estimated parameters are the variances and covariances. You don't describe the second model in sufficient detail to give any suggestion. If you want a more specific answer, you need to write out in details your models: both priors and likelihoodstatistics. Note that BAM (z Bayesian Model Averaging) only makes sense in the framework of Bayesian statistics: if you cannot cast your model in the form of a likelihood and a prior (which could depend on other priors: hierarchical model), there isn't much point in applying BAM. $\endgroup$ – DeltaIV Nov 3 '16 at 21:44
  • $\begingroup$ thanks. I think I start to get it. The other model is also using the covariance. But instead of the variances of the historical returns and the market index returns, this second model is using so called "implied variances" which are derived from an entirely other model. This other model is a differential model which can be solved analytically. So actually $θ=(θ_1,…,θ_d)$ is different for both models. $\endgroup$ – Plazi Nov 4 '16 at 9:15
  • $\begingroup$ I suspected that much. However, one of the nice things about Bayes factors is that you can use them to compare (Bayesian model selection) or combine together (BAM) nonnested models. Thus it's not a problem if the number of parameters, their priors and even the likelihoods are completely different for the two models. The real issue here is, are you able to write out the likelihoods for the two models? This is a little old, but it could be of help. $\endgroup$ – DeltaIV Nov 4 '16 at 12:50
0
$\begingroup$

After stressful days I think I have found a idea here:

http://www.immagic.com/eLibrary/ARCHIVES/GENERAL/WIKIPEDI/W120607B.pdf

Therefore the BIC of a specific model $i$ can be approximated by

$BIC = n*ln(s_e^2) + k*ln(n)$

with

$n$ : number of observations

$s_e^2$ : error variance

$k$: the number of free parameters

now this BIC can be used to derivate the Posterior Model Probabilities (weights) for the case that the models have uniform distributed priori weights

Improve this answer
$\endgroup$
  • $\begingroup$ hmmm. Note that this simplified expression for the BIC is only valid if "model errors or disturbances are independent and identically distributed according to a normal distribution and that the boundary condition that the derivative of the log likelihood with respect to the true variance is zero". I don't understand what the sentence after "boundary condition" means, but the first sentence indicates that this approximation is only correct for a i.i.d. errors with Gaussian distribution. I really don't think this is the case for time series modeling.... $\endgroup$ – DeltaIV Nov 3 '16 at 18:26
  • $\begingroup$ ...if you want to use BIC, and you don't have iid Gaussian errors, then you should use the original expression with the log-likelihood of the MLE of $\theta$. Note that even the usual expression is just an asymptotic approximation valid when $T \gg d$. $\endgroup$ – DeltaIV Nov 3 '16 at 18:29
  • $\begingroup$ Understood. But what is my $θ$ here? In the one model I am using historical returns of a stock and historical returns of a market index and calculate the quantity of interest $y$ (assuming to be the forecast for the next period) based on the covariance and variances of these... $\endgroup$ – Plazi Nov 3 '16 at 19:26
  • $\begingroup$ Ok. Now I got what my $θ$ is, namely the covariance and the two variances, which are used to compute $y_{T+h}$. I have just one missing point left regarding the original BIC: How to calculate/estimate the MLE of $θ$ for the first model? I am completely confused by the literature. All I have is the realized observations $y_{1:T}$, the estimations of the model 1 $y*_{1:T}$ in the training period as well as the new $y_{T+h}$ estimated by model 1 , and of course the values of the parameters $θ$ for each $1:T$ $\endgroup$ – Plazi Nov 4 '16 at 13:47
  • $\begingroup$ For me it's impossible to answer without seeing the model. Either you write out the model equations and assumptions in detail (which is the distribution of errors? are the observations i.i.d.? If they're not, then what's their joint distribution? Multivariate Gaussian? Dirichlet? etc.) or I can't help. $\endgroup$ – DeltaIV Nov 4 '16 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.