3
$\begingroup$

I have two families of models that can possibly explain the data at hand.

One family is rather process-based, using fairly complicated simulations and Approximate Bayesian Computation to estimate the unknown parameters of the process. If I were to remain in the ABC framework, I know how I can use different simulation models to perform model selection.

The other family of models is more descriptive and involves likelihood computations for Bayesian inference. Again, if I were to remain in this framework, I know I can compare models using Bayes factor and model posteriors for model selection.

However, I would need to compare the two families of models (ABC estimations versus non-ABC estimations), and in that case I am much less certain: the parameters do not describe the same things at all, and the estimation methods are different too. Does it make sense to compare models between ABC/non-ABC estimations using BIC or is it unsound?

I am not a statistician, so do not hesitate asking me to clarify, I will try my best!

$\endgroup$
5
  • 1
    $\begingroup$ You can always use ABC with a closed-form likelihood model. In that case, the natural distance is the absolute difference between the log-likelihoods. On the opposite, you cannot use BIC with intractable likelihood functions (and, besides, BIC is not a Bayesian procedure as it does not involve the prior or posterior). $\endgroup$
    – Xi'an
    Jan 24, 2022 at 8:00
  • $\begingroup$ @Xi'an thank you for your answer! My bad, I said BIC for ABC but I meant Bayes factor, I corrected that in the question. I am not sure to understand what you are advising to do? $\endgroup$
    – WaterFox
    Jan 24, 2022 at 17:38
  • 1
    $\begingroup$ Sorry for being unclear: if ABC is needed for at least one model, use ABC for all models. $\endgroup$
    – Xi'an
    Jan 24, 2022 at 20:45
  • $\begingroup$ Oh I see! Thank you for clarifying, it makes more sense to use ABC for both if I want to compare the model posteriors. I will unfortunately not have time to implement the simulator for the other model :'( $\endgroup$
    – WaterFox
    Jan 24, 2022 at 21:18
  • 1
    $\begingroup$ And I take back my suggestion to use the likelihood as a natural distance as the summaries and the distance should both remain the same across models. $\endgroup$
    – Xi'an
    Jan 25, 2022 at 5:48

1 Answer 1

1
$\begingroup$

Brain-storming ideas here (not 100% sure on either):

  • There is a modern ABC method called SNLE (Sequential Neural Likelihood Estimation) - which will give you an approximate likelihood (an actual analytical form of it - either using Mixture Density Networks, or Normalizing Flows). You can then maybe compare that with the theoretical model using Bayes factor.

  • Another possibility is to use the samples generated from the ABC, which are essentially samples from the posterior $p(\theta|x_{obs})$. Choose the mode of this posterior (e.g., using KDE or some other methods), which I can interpret as "this is the most likely parameter to have generated the data" - then feed it to the simulator to get many samples of $x_{i,sim}(\theta_{mode})$, which should be a representing samples of the underlying likelihood $p(x|\theta_{mode})$. Now you can maybe compare this ABC empirical likelihood to the theoretical likelihood given by your theoretical model (maybe using KDE and Bayes factor? Or maybe using KDE and comparing the resulting density to the theoretical model density).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.