0
$\begingroup$

Naïve question:

I would like to use Bayesian framework for model selection. I have more than 10 models with the same number of parameters (just different assumptions on underlying parameters of the distribution). How to apply, for example, Bayes factors in this case? One-vs-all? So

$H_0 = H_0$,

$H_A = H_1 \cup H_2 \ldots \cup H_n$,

calculate Bayes factors for $H_0$ and $H_A$ for all hypothesis and than choose the model with the highest Bayes factor as the most appropriate (or decide that none of the models are suitable)?

(I would use BIC/AIC if I would have models with different sets of parameters, but I do not know any method for this problem. Maximum likelihood does not use all the information so it is not a solution).

Here is the example (without priors). I would say that Posterior Probability (pp) of 0.58 is not big enough to make a decision in favour of second model and I would increase the sample size, but using ML I would choose third model (while second is true). Is the suggested procedure correct?

 > dataset <- rnorm(10, mean=1.0)
 > 
 > first_mu = 0.0
 > second_mu = 1.0
 > third_mu = 2.0
 > first_likelihood = prod((dnorm(dataset, first_mu)))
 > second_likelihood = prod((dnorm(dataset, second_mu)))
 > third_likelihood = prod((dnorm(dataset, third_mu)))
 > first_likelihood
[1] 1.370522e-10
 > second_likelihood
[1] 2.122489e-06
 > third_likelihood
[1] 1.492313e-06
 > pp1 = first_likelihood / (first_likelihood + second_likelihood + third_likelihood)
 > pp2 = second_likelihood / (first_likelihood + second_likelihood + third_likelihood)
 > pp3 = third_likelihood / (first_likelihood + second_likelihood + third_likelihood)
 > pp1
[1] 3.791275e-05
 > pp2
[1] 0.5871438
 > pp3
[1] 0.4128183

If I would have dataset from dnorm(n, mean=0.5), I would choose one model using likelihood, but I can choose none of models using BFs, that's why I don't want to use ML methods.

UPD: the similar situation is described here, but I still can not figure out how to switch to BFs for several models...Can not understand how posterior probability be interpreted (again, some arbitrary threshold on posterior probability? like level of significance for hypothesis testing?)

UPD2: If there are more than two models to compare then we choose one of them as a reference model and calculate Bayes factors relative to that reference. - that is even more strange...

$\endgroup$
  • 1
    $\begingroup$ AIC uses the number of parameters AND the maximum likelihood (ML), so it is the ML that will be the tie breaker. Why not use it? What do you mean by "Maximum likelihood does not use all the information so it is not a solution"? $\endgroup$ – Salmonstrikes Feb 22 '16 at 12:24
  • $\begingroup$ The model can have the highest likelihood, but BF of the model can be slightly bigger than 1 (so I should increase the sample size in this case). Also I have prior probabilities of different hypothesis, do not know how to use AIC with priors. $\endgroup$ – German Demidov Feb 22 '16 at 12:29
3
$\begingroup$

If you have $m$ models to compare, you can always produce $m$ evidences or marginal likelihoods for those models $$\mathfrak{e}_i=\int_{\Theta_i} f(x|\theta)\text{d}\pi(\theta)$$ and rank those models according to the evidences $\mathfrak{e}_i$.

Unless you have a fundamental reason or loss function to rank the models a priori and a preference to select the first one over the second, and so on, I see no reason in opposing one model $\mathfrak{M}_1$ to the conjunction of all others. And keeping this rule to evaluate all models is incoherent as the denominator should always be the same.

If you worry about the strength of a decision based on the ordering of evidences, you can check the error rates of this decision rule by simulating its performances under the posterior (or prior) predictives under all models.

$\endgroup$
  • $\begingroup$ Thank you for the answer. Yes, I can rank likelihoods, but there are several issues that I have described: likelihood tells me that a hypothesis is the most appropriate among alternatives, but only likelihood ratio can tell me is one of the hypothesis is strongly preferred than any other. The rule is just an application of Bayes factors: we have parameter space and two hypothesis that covers all space and we need to answer the question: how big is the evidence in favour of $H_i$ against all alternatives $H_A$? $\endgroup$ – German Demidov Feb 22 '16 at 13:55
  • $\begingroup$ I disagree that (a) likelihood ratios give you a notion of strength and (b) you need to always compare one hypothesis to all others: the proper loss function leads you to select the most likely model which is exactly equivalent to the ranking of evidences. $\endgroup$ – Xi'an Feb 22 '16 at 14:04
  • $\begingroup$ a) They give, it was described a lot of times (i.e. docs.lib.purdue.edu/cgi/…), b) I do not need just most likely model, I need to understand how strong is the evidence in favour to the most likely model. May be all my models are wrong, BF will show it, while likelihood will not. I.e., nicebrain.files.wordpress.com/2015/04/figure-4.png , but one of the models will have higher likelihood here for sure. Or may be I do not understand you...=( $\endgroup$ – German Demidov Feb 22 '16 at 14:13
  • $\begingroup$ here it explained better: en.wikipedia.org/wiki/Bayes_factor#Interpretation $\endgroup$ – German Demidov Feb 22 '16 at 14:15
  • $\begingroup$ Your reply was helpful. I hope I found the solution: stat.purdue.edu/docs/research/tech-reports/1993/tr93-43c.pdf . The problem is my lack of explanation skills =( sorry. $\endgroup$ – German Demidov Feb 22 '16 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.