Timeline for Are there any examples where Bayesian credible intervals are obviously inferior to frequentist confidence intervals
Current License: CC BY-SA 2.5
45 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 30, 2011 at 12:55 | comment | added | probabilityislogic | I think something which is perhaps been overlooked in the discussion above (including by me) is that the ML solution is exactly equal to the maximum of the joint posterior density using a uniform prior (so $p(\theta|X)\propto p(X|\theta)$ ($\theta$ is the vector of parameters). So you cannot claim that ML is good and Bayes is not, because ML is mathematically equivalent to a Bayesian solution (flat prior, and 0-1 loss function). You need to find a solution which cannot be produced using Bayesian methods. | |
| Jan 22, 2011 at 13:25 | comment | added | Dikran Marsupial | @Joris, Figure 18.7 on page 305 just shows that using an informative (not uninformative) prior, the maximum likelihood estimate lies outside the Bayesian credible interval. There is nothing in the least surprising about that. As has already been pointed out, a flat prior on branch length is unlikely to be uninformative (transformation groups), especially when needlessly truncated (it is possible to use improper priors). | |
| Jan 21, 2011 at 23:10 | comment | added | Joris Meys | @probabilityislogic : But again, most studies rigthfully conclude that they can't be compared. And I still follow Felsenstein that, in case no further knowledge is available, the risk on bias is far larger with a bayesian than with an ML estimate for a phylogenetic tree. If you dive into the literature on phylogeny ( and check the papers that are not online as well, science didn't start in 1998), you'll see that this controversy has been debated heavily for the past 50 years. You and @Dikran might disagree, but the comments here are far from the right place to discuss this properly. Cheers | |
| Jan 21, 2011 at 23:04 | comment | added | Joris Meys | @probabilityislogic : There have been numerous frameworks built up by now around bayesian posterior probabilities as alternative for bootstrap support values, but most of the studies conclude - rightfully - that both cannot be compared. And for the estimates of the prior both birth-death processes (data-based) as theoretical distributions for branch lengths have been used extensively. Bayesian applications like mrBayes can reduce calculation time significantly, but discussion remains whether they perform better or worse, each side of the argument bringing "proof" for the claim. | |
| Jan 21, 2011 at 23:00 | comment | added | Joris Meys | @probabilityislogic : we're talking about trees. The likelihood of the tree is the multiplication of all likelihoods at each site (node) of the tree, which is defined as the sum over all possible nucleotides that may have existed at the interior nodes of the tree, of the probabilities of each scenario of events. And that probability is defined by a model which involves T (or u), the Jukes-Cantor model being the most easy one. As said, phylogeny does not fit into any classical framework. | |
| Jan 21, 2011 at 22:55 | comment | added | Joris Meys | @Dikran : the graph about the truncation of T is shown on page 305 (fig 18.7) | |
| Jan 21, 2011 at 16:08 | comment | added | probabilityislogic | I'm a bit curious, how does the ML solution work for $t$ if you just plug in $P$ into your likelihood. The derivative will be (by chain rule) $\frac{dL}{dt}=\frac{dL}{dP}\frac{dP}{dt}=0$ but from the function for $P$ this means $\frac{dP}{dt}=\frac{4u}{3}e^{-\frac{4}{3}ut}$, so setting $u \rightarrow 0$ and $t\rightarrow\infty$ such that $P$ is unchanged (and equal to $P_{MLE}$) would solve the ML equation? Or is there something about $u$ which is not stated in the information? | |
| Jan 21, 2011 at 10:56 | comment | added | probabilityislogic | That the solutions are different is no more and no less surprising than if you used a different model between P and t. | |
| Jan 21, 2011 at 10:55 | comment | added | probabilityislogic | @joris I can understand what you are trying to say, in setting a prior you are describing a state of knowledge, just as if you are setting a sampling distribution. Now in the uniform prior on $P$ you are describing a state of knowledge that it is possible for "no change" and "one or more changes" to occur on a given branch. Probability theory tells you how to coherently transform this into the same state of knowledge about $t$, given your knowledge about the relationship between $P$ and $t$. So a "flat" prior for $t$ necessarily is describing a different state of knowledge. | |
| Jan 21, 2011 at 10:32 | comment | added | Dikran Marsupial | @Joris "the whole point Felstenstein makes is that t and u are linked. Meaning that a flat prior on t gives a greatly biased prior on u and vice versa." It may be that this bias is what you get if you make a minimally informative prior that includes the prior knowledge that the units of measurement should have no effect on the conclusion (transformation groups). | |
| Jan 21, 2011 at 10:29 | comment | added | Dikran Marsupial | @Joris, it is a while since I read the chapter in question, but IIRC Felseneteins problem was that a flat prior on branch length is biologically implausible. I agree, but a flat prior on branch length is not necessarily an uninformative prior. Felsensteing seems to think (incorrectly) that only flat priors are uninformative, and hence isn't aware of other choices that may be uninformative and biologically plausible. I should point out though that if you have knowledge of what is and what isn't biologically plausible, then you are not entirely uninformed, and neither should be your prior! | |
| Jan 21, 2011 at 10:29 | comment | added | Joris Meys | @probabilityislogic : the whole point Felstenstein makes is that t and u are linked. Meaning that a flat prior on t gives a greatly biased prior on u and vice versa. You'll have to use a prior that favorizes certain values for either of it in order to have a prior that actually makes biologically sense. So you need to know at least something about either the transformation rate or the mutation time to use eg mrBayes in phylogeny. | |
| Jan 21, 2011 at 10:21 | comment | added | Joris Meys | @Dikran : I'll look it up tonight. It's where he demonstrates the effect of the truncation on the t prior. Actually, it's almost a page big, you should have seen it when you read the chapter. It's pretty the center of his story... | |
| Jan 21, 2011 at 10:20 | comment | added | Joris Meys | comment removed - whatever... | |
| Jan 21, 2011 at 10:16 | comment | added | Dikran Marsupial | @Joris, can you give a specific page number? | |
| Jan 21, 2011 at 10:11 | comment | added | Dikran Marsupial | @probabilityislogic - I have Felsenstein's book, unfortunately his reasoning is faulty as he seems to think that all flat priors are uninformative and vice-versa and thus considers the fact that two flat priors on different parameterisations of the same thing give different conclusions is an indication there is a problem. The premise is wrong, and the conclusion unsurprising to anyone familiar with the idea of transformation groups. Essentially an uninformative prior on branch length should be insensitive to the choice of units, which would give a prior that was flat on a logarithmic scale. | |
| Jan 21, 2011 at 3:08 | comment | added | probabilityislogic | Apologies (again), I wrote the fraction incorrectly (today is just not my day!). So it should be that you can write $P=Pr(Y<\frac{4u}{3})$ where $Y \sim Expo(t)$ so that $E(Y)=\frac{1}{t}$. If we do not observe $u$ or $t$ then the model is not identifiable (i.e. there are an infinite amount of $u$ and $t$ values which give the same $P$). | |
| Jan 20, 2011 at 23:17 | comment | added | probabilityislogic | @Joris Meys - I do appreciate the reference to the book (but its seems as though without a link, I am to buy his book in order to read your reference), which is where all the arguments are. The equation you presented for the model is simple enough (0<P<1, t>0, u>0 with a relation between each), in fact it could be expressed as the $P=Pr(Y<u)$ where Y has an exponential distribution with rate parameter $t$ (or mean $\frac{1}{t}$). The CI needs to be around one of the parameters defining the model you describe. The interpretation will depend on the context, as you correctly point out. | |
| Jan 20, 2011 at 9:34 | comment | added | Joris Meys | @probabilityislogic : To show the difference in nature of the problem : you talk about confidence intervals. Now try to define a confidence interval around a phylogenetic tree... | |
| Jan 20, 2011 at 9:33 | comment | added | Joris Meys | @probabilityislogic : the whole discussion revolves around Felstensteins claim that it is impossible to put a prior without making impossible assumptions about either time or mutation rate. Remember we're talking about phylogenetic trees. This concept makes for quite a different framework, as it's not a classical equational model in a space of real numbers. I'd suggest you read the chapter of his book to see his argument on how under certain condition the Bayesian approach can be proven to be wrong. I'd like to stress this is ONE example. It doesn't say anything about Bayesian in general. | |
| Jan 20, 2011 at 7:17 | comment | added | probabilityislogic | @joris meys - I understand that you did give a reference, but your discussion does not talk about how the confidence interval solution is superior to the Bayesian credible interval. This means that basically the confidence interval basically needs to be uncalculable using Bayesian methods. By showing the Bayesian solution which gives the same interval, you can show what prior information was implicitly contained in the procedure to generate the confidence interval. | |
| Jan 19, 2011 at 9:10 | comment | added | Joris Meys | @probabilityislogic : I gave the references. This is a discussion site, not a scientific journal. Please read the comments and the reference I gave for more information. and before you call it a poor example. | |
| Jan 19, 2011 at 6:03 | comment | added | probabilityislogic | This is a very poor example of the "inferiority" of Bayesian methods, of exactly the same type Jaynes speaks of in his 1976 paper. You need to write down what the numerical/mathematical equation that the ML (or other frequentist method) does, and the corresponding Bayesian method and its numerical answer! You have written down the model, but no solution to the estimation of anything to do with it! The rest of your answer would be greatly improved if you wrote down what the frequentist answer using ML actually is. | |
| Nov 19, 2010 at 17:32 | comment | added | Dikran Marsupial | I am not suggesting a Bayesian approach is any better than a frequentist one - horses for courses. In this case it is probably transformation groups that hold the key. It is quite possible that a prior on branch length that is invariant to the units used is equivalent to a flat prior on the probability of a change - in which case Felsensteins criticism is badly misguided. Uninformative priors are not necessarily flat and it is inappropriate to criticize uninformative priors without mentioning the standard procedures for finding them! Not that this means Bayesian is better, of course. | |
| Nov 19, 2010 at 13:32 | comment | added | Joris Meys | @Dikran: entropy maximization without testable information gets only one constraint: probabilities sum up to one. Most often the uniform distribution is taken there. I don't take it as granted, but I do agree with Felsensteins calculations and reasoning. So we disagree, like more people in that field. Felsenstein if far from accepted by everybody, and I'm not accepting everything he says. But on this point, I follow him. Sometimes a Bayesian approach is not superior to another one. And the case he describes is one such a case according to me. YMMV. | |
| Nov 19, 2010 at 13:06 | comment | added | Dikran Marsupial | I was recently given a copy of Felsenstein's book. In chapter 18 he does not say why you can't use an improper flat prior on 0-infinity. Neither does he mention MaxEnt or transformation groups in his criticism of uniformative priors. While the rest of the book may be very good; this suggests inadequate scholarship on that particular issue. Caveat lector - just because something appears in a text book or journal paper, doesn't mean that it is correct. | |
| Sep 6, 2010 at 16:22 | comment | added | Dikran Marsupial | @Joris - it may be that an uninformative prior cannot be constructed in this case, but nothing that you have written so far establishes that to be the case. What does Felsenstein write about MAXENT and transformation groups (the two main techniques used to determine an uninformative prior for a particular problem)? If he has not investigated those methods, how can he know an uninformative prior is impossible? It looks to me that a flat prior on p corresponds a flat prior on log(t), which is a well known Jeffreys' prior. Can you demonstrate that the flat log(t) prior is informative? | |
| Sep 6, 2010 at 16:03 | comment | added | Joris Meys | @Dikran : The point is not whether a flat prior is uninformative. The point is that an satisfying uninformative prior cannot be defined due to the nature of the model. Hence rendering the whole method unusable if you don't have prior information, and thus leading to the conclusion that Bayesian inference in this case is inferior to the ML approach. Felsenstein never said a flat prior was uninformative. He just illustrated why an uninformative prior cannot be determined, using the example of a flat prior. | |
| Sep 5, 2010 at 23:09 | comment | added | Dikran Marsupial | @Joris, as I said a flat prior is NOT NECCESSARILY UNINFORMATIVE. Consider this, if two priors give different results, then the must logically represent a different state of prior knowledge (see early chapters of Jaynes book that set out desiderata for Baysian inference). Therefore the "flat p" prior and "flat t" prior cannot both be uninformative. Felsenstein may be an expert on phylogenetic inference, but it is possible that he is not an expert on Bayesian inference. If he states that two priors giving different results are both uninformative, he is at odds with Jaynes (who certailny was). | |
| Sep 5, 2010 at 20:02 | comment | added | Joris Meys | @Dikran : Guess you are missing the point : using the same uninformative prior gives two different models with Bayesian statistics on the same dataset. Not so with ML. The Bayesian can be very biased due to the nature of the model and the incompatibility of that model with infinite priors. You don't have to believe me. Felsenstein is the authority on phylogenetic inference, and his book explains you better than I will be able to. Reference in a previous comment. | |
| Sep 4, 2010 at 8:49 | comment | added | Dikran Marsupial | @Joris, also in your original comment you suggest the flat prior on t must be truncated, because otherwise the area under the density function is infinite. This is not true, there are plenty of problems where improper priors work very well, so there is not necessarily a need to truncate the flat prior. | |
| Sep 4, 2010 at 8:39 | comment | added | Dikran Marsupial | Joris, I think you are missing the point, a flat prior is not necessarily non-informative. It is completely reasonable for the same state of knowledge/ignorance to be expressed by a flat prior on p and (say) a flat prior on log(t) (which is a very common Jeffrey's prior) rather than a flat prior on t. Does the book investigate ideas of MAXENT and transformation groups for this problem? There isn't enough detail in your example, but from what I can tell even a truncated flat prior on t is likely to be inconsistent with prior knowedge about t. | |
| Sep 4, 2010 at 0:11 | comment | added | Joris Meys | @Dikran : Regarding the flat t-prior: the prior gets truncated. When truncated at 5(!), most of the mass of the prior on p is concentrated around the maximum p-value. With larger truncation values, this effect is even more pronounced. The point is-again- that it's impossible to find a sensible prior when you have no prior knowledge in phylogenetic inference. | |
| Sep 4, 2010 at 0:08 | comment | added | Joris Meys | @Dikran : it's not a problem. It is a fact. The problem is that p and t are strictly related, and hence should give exactly the same model. That happens in an ML approach, but that doesn't happen in the Bayesian approach. In Felsensteins example, a truncation of the t-prior at 700 or larger makes that the credible interval doesn't cover the true value any more. In this particular case, i.e. the lack of prior knowledge, Bayesian inference just isn't feasible. There is no sensible "uninformative" prior that can be used. | |
| Sep 3, 2010 at 23:45 | comment | added | Dikran Marsupial | It isn't clear to me why it is a problem that a flat prior on p implies an exponentially decreasing prior on t. If that is inconsistent with biological knowledge, it simply means that a flat prior on p does not reflect actual prior knowledge. I also don't see why it is a problem to use an improper flat prior on t (other than I would have thought it inconsistent with prior knowledge; the branch time can't be say a billion years, if it were we wouldn't be here yet, so it is inappropriate to use a flat prior). Note that flat priors don't necessarily imply ignorance. | |
| Sep 3, 2010 at 23:04 | comment | added | Joris Meys | @Srikant : I tried to clarify a bit more. Actually, P is the parameter that is used in the likelihood function for optimization, and the formula merely gives its relation to t, which can alternatively be used in the likelihood function. Sorry I can't be more clear. If Phylogeny interests you, I can surely recommend Felsensteins book, it's a gem. sinauer.com/detail.php?id=1775 | |
| Sep 3, 2010 at 23:00 | history | edited | Joris Meys | CC BY-SA 2.5 |
added 505 characters in body
|
| Sep 3, 2010 at 22:44 | comment | added | user28 | I guess I just wanted you to clarify what variables in your equation was data and which ones were the parameter as it was not clear from your post especially to someone like me who is an outsider. I am still lost but I guess I would need to read the book to find out more. | |
| Sep 3, 2010 at 22:29 | comment | added | Joris Meys | @Srikant: The example in Felsensteins book is based on a Jukes-Cantor model for DNA evolution. Data is DNA sequences. You want to estimate the probability of change in your sequence, which is related to your branch length based on the mentioned formula. Branch lengths are defined as time of evolution : the higher the chance for changes, the more time that passed between the ancestor and the current state. Sorry, but I can't summarize the whole theory behind ML and Bayesian phylogenetic inference in just one post. Felsenstein needed half a book for that. | |
| Sep 3, 2010 at 22:24 | history | edited | Joris Meys | CC BY-SA 2.5 |
added 2 characters in body
|
| Sep 3, 2010 at 22:23 | comment | added | user28 | Could you clarify what is your data and what are the parameters you would be estimating in your model? I am a bit confused on this point. Also, could you please use $$ instead of $ to center the formula? The font size is very small right now. | |
| Sep 3, 2010 at 22:22 | comment | added | Joris Meys | Jaynes first example: Not one statistician in his right mind will ever use an F-test and a T-test on that dataset. Apart from that, he compares a two-tailed test to P(b>a), which is not the same hypothesis tested. So his example is not fair, which he essentially admits later on. Next to that, you can't compare "the frameworks". What are we talking about then? ML, REML, LS, penalized methods,...? intervals for coefficients, statistics, predictions,...? You can as well ask whether Lutheran service is equivalent or superior to Shiite services. They talk about the same God. | |
| Sep 3, 2010 at 22:16 | history | edited | Joris Meys | CC BY-SA 2.5 |
added 373 characters in body
|
| Sep 3, 2010 at 20:52 | comment | added | Dikran Marsupial | It is possible to be interested in the differences between Bayesian and frequentist statistics without it being a quarrel. It is important to know the flaws as well as benefits of ones preferred approach. I specifically excluded priors as that is not a problem with the framework, per se, but just a matter of GIGO. The same thing applies to frequentists statistics, for example by assuming and incorrect parametric distribution for the data. That wouldn't be a criticism of frequentist methodology, just the particular method. BTW, I have no particular problem with improper priors. | |
| Sep 3, 2010 at 20:24 | history | answered | Joris Meys | CC BY-SA 2.5 |