6
$\begingroup$

I have some models based on stochastic differential equations (SDEs). Because of the definition of these models, I can simulate data, but I cannot compute the likelihood function / distribution function. Therefore, I currently plan to use approximate Bayesian computation (ABC) to fit the parameters of these models.

However, I also need a method to compare different SDEs, which are currently discussed as possible explanations of the data, while accounting for the complexity of the parameters. Normally, I would compare these models based on DIC, LOOIC etc, but all these require the likelihood to be known.

Is there any method for comparing the model complexity, if the likelihood is unknown?

The only way I could think that might work, is to use a Bayesian model selection (i.e. using a categorial variable to switch between the models), but I am not sure if this would work at all.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Is (cross-validated) out of sample prediction accuracy an option? $\endgroup$
    – Eoin
    Commented Oct 21, 2020 at 13:34
  • 2
    $\begingroup$ i haven't had a chance to read this article, but this might be relevant.. ill take a look in a few hours. pnas.org/content/108/37/15112 $\endgroup$
    – fool
    Commented Oct 22, 2020 at 0:36
  • 1
    $\begingroup$ @user228809 Thank you for the link. I've only read the abstract so far, but that seems to indicate that this is just not possible, or at least not well-founded due to unknown information loss. What a pitty... $\endgroup$
    – LiKao
    Commented Oct 22, 2020 at 7:29
  • $\begingroup$ @Eoinisonthejobmarket I was also thinking about that, but I am not sure how that would work in this case. I am not sure if just drawing from the (unknown) distribution would be sufficient as a prediction, or if I have to add something. Also, I might not have enough data to do cross-validation. $\endgroup$
    – LiKao
    Commented Oct 22, 2020 at 7:31
  • $\begingroup$ You said you can simulate data though? Isn't that a prediction? Also, if the data is small (and model fitting doesn't take too long), leave-one-out cross validation maybe? $\endgroup$
    – Eoin
    Commented Oct 22, 2020 at 8:04

1 Answer 1

Reset to default
4
$\begingroup$

There are examples in the ABC literature of model selection through Bayes factors. An ecological individual based model example is here:
https://doi.org/10.1016/j.ecolmodel.2017.07.017
The paper involves picking between models of different complexity so hopefully it will be useful even if it doesn't deal with SDEs directly.

However, I should also mention that there are those that think that Bayes factors (or odds ratios or any other kind of Bayesian model selection) is just philosophically unsound. This is an argument for example forcefully stated by Gelman and Shalizi.

http://doi.org/10.1111/j.2044-8317.2011.02037.x

Their argument is that models ought to be judged primarily by their simulation output. Rather than a formula then you ought to come up with a series of model checks where you compare simulation outputs with either data you held out or other features of the data that weren't directly part of the fitting (in your case, since it's ABC, that would mean other summary statistics that were not included in the rejection step).
In my view this gives a much better grip at what the model is good and bad and is a lot more convincing than citing some obscure probability measure. Of course though, this is a very context-dependent exercise.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I have yet to read the Gelman paper, but if I understand your summary correctly, this shows my attempt is kind of futile. I wanted to test a more advanced model vs. a simpler one, but the more advanced model was specifically designed to capture features of the data, the simpler one cannot represent. I am unsure if this ability is due to higher complexity (more parameters), or if the more advanced model is doing some kind of overfitting, allowing it to capture these features. But I guess I have to understand the question better. $\endgroup$
    – LiKao
    Commented Oct 28, 2020 at 8:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.