Timeline for Is it inherently invalid to use BIC for model averaging?
Current License: CC BY-SA 3.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 18, 2016 at 18:08 | comment | added | Bryan | Let us continue this discussion in chat. | |
| Mar 18, 2016 at 17:48 | comment | added | Glen_b | en.wikipedia.org/wiki/Posterior_predictive_distribution | |
| Mar 18, 2016 at 17:35 | comment | added | Bryan | What is the posterior predictive distribution? Is that a 20-dollar word way of saying "the distribution of the model weights". Or are the "mixture weights" something altogether different? Eventually, the jargon reaches the point where I am tempted, as a biologist, to just throw up my hands and go back to uncritically hunting p values. And how does one have any inkling at all if one has the "true" model in the candidate set? Is it all just farting in the dark, as I said? | |
| Mar 18, 2016 at 17:21 | comment | added | Glen_b | I don't understand your question. What does "that" refer to in particular, and what do you mean by "distribution of weights"? | |
| Mar 18, 2016 at 17:16 | comment | added | Bryan | Is that the distribution of weights? | |
| Mar 18, 2016 at 15:21 | comment | added | Glen_b | You work out approximate posterior probabilities of each model in the usual model-averaging way (by choosing them proportional to $e^{-(\Delta_k)/2}$ where $\Delta_k=\text{BIC}_k-\text{BIC}_0$. You then work out the posterior predictive distribution as a mixture of the conditional (on each model) ones, where the mixture weights are the posterior probabilities. One always makes some presumptions, and obviously our inferences are conditional on those. | |
| Mar 18, 2016 at 13:57 | comment | added | Bryan | Secondly, if one uses model averaging, and one is after that 95% posterior interval, is that measured by deltaBIC, by proportion of total weights, or by some other means? Does everything go out the window if one pre-constrains possible models based on some presumptions? | |
| Mar 18, 2016 at 13:54 | comment | added | Bryan | So, how does one in the real world know whether or not one actually has the "true" model in your set of models? Or is it all just farting in the dark? If one has no way to know, then does one choose efficiency (AIC) over consistency (BIC)? | |
| Mar 17, 2016 at 6:36 | comment | added | Glen_b | @Richard It may not be so bad. If the class of models is a sufficiently flexible one, it may include models that can come arbitrarily close to the true one ... at the expense, perhaps, of being more flexible than really needed (larger variances with nonparametric regression when trying to fit a linear relationship for example). That doesn't cover all possibilities (such as missing important predictors, for example), but it does at least suggest there's not necessarily reason to worry over-much about functional-form. | |
| Mar 17, 2016 at 6:15 | comment | added | Richard Hardy | thank you for the explanation. Since having a true model within the candidate model set may have zero probability in many applied cases, the picture does not look bright (at least in theory). But I suppose in practice we do not worry about that too much. | |
| Mar 16, 2016 at 23:14 | comment | added | Glen_b | ctd ... but results would be hard to generalize -- one thing to do would be to try to see just how bad it could get. | |
| Mar 16, 2016 at 23:12 | comment | added | Glen_b | @RIchard It's a big question if we don't constrain it -- even as a new question I think it's too broad. As far as I can see, nothing prevents even the best members of your candidate model class from being arbitrarily bad at prediction if they don't contain the true model. It may be possible to say something about bounds that relate to some suitable distance measure between the true model and the nearest one in the class. I may have seen something like that but off the top of my head don't recall any such results. Alternatively, one could take a simpler tack and present some examples; ... ctd | |
| Mar 16, 2016 at 19:17 | comment | added | Richard Hardy | We had some discussion with the OP on how important it is to have the true model within the set of candidate models. So my extra question is, how bad is it if the true model is not in the set of candidate models? What property of BIC-weighting breaks then? Do BIC-weight become useless? If not, what can we still claim? | |
| Mar 16, 2016 at 3:47 | history | edited | Glen_b | CC BY-SA 3.0 |
added 141 characters in body
|
| Mar 16, 2016 at 2:20 | history | answered | Glen_b | CC BY-SA 3.0 |