Timeline for Cross-validation vs empirical Bayes for estimating hyperparameters
Current License: CC BY-SA 3.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 7, 2012 at 17:30 | comment | added | Dikran Marsupial | @probabilityislogic - note the model assumptions includes the prior as well as the likelihood. I'm not saying CV is necessarily better than marginal likelihood. For example for a GP, if the covariance function were inappropriate, marginal likelihood maximisation will choose the hyper-parameters assuming that the covariance function is right, but we have an odd sample of data from that prior, but cross-validation will choose hyper-parameters that compensate somewhat for this in order to maximise performance on the validation folds. | |
| Mar 22, 2012 at 21:59 | vote | accept | Memming | ||
| Mar 19, 2012 at 11:12 | comment | added | Dikran Marsupial | @probabilityislogic, I'm not quite sure what you are getting at (problem undoubtely at my end! ;o). I can tell you from practical experience though that the issue is very real. I have been working on problems in model selection for several years, and I have come across many problems where maximising the marginal likelihood turns out to be a very bad idea. Cross-validation performs about as well for most datasets, but where it performs badly it rarely performs catastrophically as evidence maximisation sometimes does. | |
| Mar 19, 2012 at 11:09 | comment | added | Dikran Marsupial | Choosing the hidden layer size is a clear example of the $\theta$ in the original question, where the weights the $\phi$ have been marginalised. In that case, choosing the hidden layer size by maximising the marginal likelihood works very badly on some datasets. | |
| Mar 19, 2012 at 11:06 | comment | added | Dikran Marsupial | @Neil G, there is a closed form expression for the cross-validation error of many models as well (e.g. Gaussian Processes). As I said, it isn't the same indicator of generalisation performance, as it depends on the stochastic model (as Wahba puts it). You gan have two Bayesian models that give the same predictions everywhere, but which have very different marginal likelihoods, but they would have assympotically similar cross-valdiation estimates. | |
| Mar 19, 2012 at 10:19 | comment | added | probabilityislogic | Regarding robustness of cv - I though cv was still sensitive to the choice of measure for the loss function. This is mathematically the same thing as the evidence being sensitive to choice of likelihood function, isnt it? it seems more like "conceptual robustness" rather than anything real. | |
| Mar 19, 2012 at 9:52 | comment | added | Neil G | Spline models and neural network are good examples where you can't calculate a likelihood over all of the parameters (for example, degree of the spline, neural network size and number of connections) and so those are examples where cross-validation gives you the ability to make good choices for those parameters. Maybe there's a hole in my understanding because I don't see what exactly is different about those parameters. What do you think? | |
| Mar 19, 2012 at 9:50 | comment | added | Neil G | He can get the same "indicator of generalization performance" by checking the total log-probability of the data given the estimated distribution returned by EB (which will be equal to the entropy of that distribution). There's no way to beat it in this case because it is the analytical solution to this problem. I don't see why cross-validation would makes sense when you can calculate a likelihood for EB. | |
| Mar 19, 2012 at 9:49 | history | edited | Dikran Marsupial | CC BY-SA 3.0 |
added 638 characters in body
|
| Mar 19, 2012 at 9:45 | comment | added | Dikran Marsupial | p.s. I have been performing an analysis of avoiding overfitting in neural networks with Bayesian regularisation where the regularisation parameters are tuned via marginal likelihood maximisation. There are situations where this works very badly (worse than not having any regularisation at all). This seems to be a problem of model mis-specification. | |
| Mar 19, 2012 at 9:44 | comment | added | Dikran Marsupial | CV is robust against misspecification in the sense that it still gives a useful indicator of generalisation performance. The marginal likelihood may not as it depends on the prior on $\phi$ (for example), even after you have marginalised over $\phi$. So if your prior on $\theta$ was misleading, the marginal likelihood may be a misleading guide to generalisation performance. See Grace Wahba's monograph on "spline models for observational data", section 4.8 (it doesn't say a great deal, but there isn't much on this topic AFAIK). | |
| Mar 18, 2012 at 0:40 | comment | added | Neil G | Why do you say that CV is robust against a mis-specified model? In his case, there is no such protection since cross-validation is searching over the same space that EB is calculating a likelihood. If his modelling assumptions are wrong, then cross-validation won't save him. | |
| Mar 17, 2012 at 13:21 | history | answered | Dikran Marsupial | CC BY-SA 3.0 |