Timeline for What to do if the rate of misclassification in Bayesian model selection depends on the model parameter
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 2, 2020 at 11:31 | comment | added | LiKao | Thank you for the additional literature. I will have a look at it. I agree that adding priors may not be a good idea, which is why I am trying to get a deeper understanding by discussing this issue here. If all else fails, I might just drop the "unclassified" part, which gives me a much better picture of the additional data. I would love to change the experimental protocol, but unfortunately, this is a re-analysis of existing data, so I am stuck with what I have. | |
| Mar 2, 2020 at 10:48 | comment | added | Camille Gontier | By the way, in case you are interested, I found some interesting literature on how to estimate the complexity and the number of free parameters in a model, see for instance: Spiegelhalter, David J., et al. "Bayesian measures of model complexity and fit." Journal of the royal statistical society: Series b (statistical methodology) 64.4 (2002): 583-639. | |
| Mar 2, 2020 at 10:48 | comment | added | Camille Gontier | Thanks a lot, I edited my answer. I indeed took the problem in the wrong way. I’m not sure that choosing a different prior is a very “lawful” way to handle the problem, but the experimental protocol (i.e. the way you obtain your data) might be used to increase the informativeness of data. Apart from the number of data points, do you have other experimental parameters you can choose? | |
| Mar 2, 2020 at 10:47 | history | edited | Camille Gontier | CC BY-SA 4.0 |
added 1188 characters in body
|
| Mar 2, 2020 at 10:09 | comment | added | LiKao | So one part I found out about this, is that this not just a kind of identifiability issue at the core, but rather that at some parameter values the data itself becomes less indicative for any model, which is bad for classification. In general, that would not be a problem, because misclassification due to lack of evidence is in fact to be expected. The problem for me arises, because the more complex model is just in the set of model to allow the MCMC to leave participants as "unclassified", but because parameters for different groups vary, this leads to unexpected artifacts. | |
| Mar 2, 2020 at 9:36 | comment | added | LiKao | The example above is just a general illustration of the effect I am observing with a much more complex model. In my actual analysis, I am using MCMC sampling with a categorical variable to actually select the model, but it is much easier to show this problem with BF. I would neither trust BIC or AIC with the actual models, because the degrees of freedom are almost impossible to estimate for these kinds of models. See also my answer here: stats.stackexchange.com/questions/313887/… | |
| Mar 2, 2020 at 9:24 | comment | added | Camille Gontier | Deciding which prior to use is indeed a subjective task which can have a strong effect on model selection. What are your results if, instead of using the Bayes Factor, you use the BIC or the AIC ? | |
| Mar 2, 2020 at 8:56 | comment | added | LiKao | I am not sure that this is actually the same effect you are observing with your model. If I understood you correctly, in your case the more complex model becomes misclassified as the simpler model, which is to be expected, as a mixture of Gaussians actually converges towards a single gaussian if the variance is increased. However, I kind of has the opposite effect, with the simpler model mimicking the more complex one. Because BF usually favors the simpler model, it should still prefer the simpler one, but it doesn't. So I still think I could somehow tweak the priors to fix this issue. | |
| Feb 28, 2020 at 17:21 | history | answered | Camille Gontier | CC BY-SA 4.0 |