This is more of a philosophical question, but from a purely Bayesian standpoint how does one actually form prior knowledge? If we need prior information to carry out valid inferences then there seems to be a problem if we have to appeal to past experience in justifying today's priors. We're apparently left with the same question regarding how yesterday's conclusions were valid, and a kind of infinite regress seems to follow where no knowledge is warranted. Does this mean that ultimately prior information must be assumed in an arbitrary way, or perhaps based on a more "frequentist" style of inference?
$\begingroup$
$\endgroup$
6
-
$\begingroup$ What about prior knowledge coming from past experiments? $\endgroup$– Christoph HanckMar 14, 2016 at 20:02
-
$\begingroup$ That's the question I'm asking. How did those experiments generate actual knowledge? $\endgroup$– dsaxtonMar 14, 2016 at 20:04
-
$\begingroup$ You yourself are the product of a stochastic probabilistic process. (Unless you have much higher prior belief in your specific divine creation than I). So I suppose, yes, possessing consciousness and having the ability to reason and incarnating as the dsaxton who chose Pi Day to ask this question can be seen as being based on something like an infinite sequence of possible repetitions over an infinite multiverse. But that's probably overthinking it. $\endgroup$– Dalton HanceMar 14, 2016 at 20:06
-
$\begingroup$ @dsaxton, for example by producing estimates of, say, the effectiveness of a drug, which we then seek to update via a new sample. $\endgroup$– Christoph HanckMar 15, 2016 at 6:54
-
$\begingroup$ Right, but does this then reduce to a kind of "cumulative" frequentism? $\endgroup$– dsaxtonMar 15, 2016 at 13:06
|
Show 1 more comment
2 Answers
$\begingroup$
$\endgroup$
7
Speaking of prior knowledge can be misleading, that is why you often see people speaking rather about prior beliefs. You do not need to have any prior knowledge to set up a prior. If you needed one, how would Longley-Cook manage with his problem?
Here is an example from the 1950s when Longley-Cook, an actuary at an insurance company, was asked to price the risk for a mid-air collision of two planes, an event which as far as he knew hadn't happened before. The civilian airline industry was still very young, but rapidly growing and all Longely-Cook knew was that there were no collisions in the previous 5 years.
Lack of data about mid-air collisions was not a problem to assign some prior to it that lead to pretty accurate conclusions as described by Markus Gesmann. This is extreme example of insufficient data and no prior knowledge, but in most real life situations you would have some out-of-data beliefs about your problem, that can be translated to priors.
There is a common misconception about priors that they need to be somehow "correct", or "unique". In fact, you can purposefully use "incorrect" priors to validate different beliefs against your data. Such approach is described by Spiegelhalter (2004) who describes how a "community" of priors (e.g. "skeptical", or "optimistic") can be used in decision-making scenario. In this case it is not even prior beliefs that are used to form priors, but rather prior hypotheses.
Since when using Bayesian approach, you include both the prior and data into your model, information from both sources will be combined. The more informative is your prior comparing to data, the more influence it would have, the more informative is your data, the less influence would your prior have.
Eventually, "all models are wrong, but some are useful". Priors describe beliefs that you incorporate in your model, they do not have to be correct. It is enough if they are helpful for your problem, as we are dealing only with approximations of reality that are described by your models. Yes, they are subjective. As you already noticed, if we needed prior knowledge for them, we would end up in a vicious circle. Their beauty is that they can be formed even when confronted with shortage of data, so to overcome it.
Spiegelhalter, D. J. (2004). Incorporating Bayesian ideas into health-care evaluation. Statistical Science, 156-174.
-
1$\begingroup$ Community of priors is a clear bastardization of Bayesian approach. Unless one has multiple personalities, there can't be multiple priors. The prior is supposed to capture your prior belief, all that you know about the phenomenon. If you have multiple priors, you'll run into even more philosophical issues than Bayesian approach already has. $\endgroup$– AksakalMar 15, 2016 at 16:50
-
1$\begingroup$ @Aksakal using them as described by Spiegelhalter is pretty appealing: use different priors and compare the outcome and check how much do they influence the outcome. Besides, they are nice example of that prior does not have to be "correct". $\endgroup$– Tim ♦Mar 15, 2016 at 17:25
-
2$\begingroup$ it's appealing technically but logically inconsistent. It basically kills the whole Bayesian statistics' basis about beliefs, subjective probabilities etc. If you have many priors, why not have infinite number of priors? In this case how is this different from frequentist approaches? Once you run infinite number of priors, your outcome will converge to pure frequentist result or uninformative prior or something along those lines. $\endgroup$– AksakalMar 15, 2016 at 17:35
-
1$\begingroup$ @Aksakal when creating single prior you "weight" the different sources of evidence "in your head" to come up with something -- how does it differ from setting multiple priors, one per source of belief? $\endgroup$– Tim ♦Mar 15, 2016 at 17:54
-
2$\begingroup$ Special thanks for the reference to Spiegelhalter 2004. I like this sentence from the closing paragraph: "The general statistical community, who are not stupid, have justifiably found somewhat tiresome the tone of hectoring self-righteousness that has often come from the Bayesian lobby." $\endgroup$– amoebaMar 20, 2016 at 23:57
$\begingroup$
$\endgroup$
9
I think you're making the mistake of applying something like the frequentist concept of probability to the foundations of the subjective definition. All that a prior is in the subjective framework is a quantification of a current belief, before updating it. By definition, you don't need anything concrete to arrive at that belief and it doesn't need to be valid, you just need to have it and to quantify it.
A prior can be informative or uninformative and it can be strong or weak. The point of those scales is that you don't have any implicit assumptions about the validity of your prior knowledge, you have explicit ones, and sometimes that can be "I have no information." Or it can be "I am not confident in the information I have." The point is, there is no requirement that prior knowledge is "valid". And that assumption is the only reason your scenario seems paradoxical.
By the way, if you like thinking about the philosophy of probability, you should read The Emergence of Probability by Ian Hacking and its sequel, The Taming of Chance. The first book especially was really illuminating in how the concept of probability came to have dual and seemingly incompatible definitions. As a teaser: did you know that until fairly recently, calling something "probable" meant that it was "approvable", i.e. that it was "approved by the authorities" or that it was a generally well respected opinion. It had nothing whatsoever to do with any concept of likelihood.
-
1$\begingroup$ Interesting statement about the meaning of "probable". Until fairly recently is until when exactly? I find 11 occurrences of "probable" in the works of Shakespeare and they seem to have the usual meaning. That's 400 years ago. $\endgroup$– amoebaMar 20, 2016 at 17:41
-
1$\begingroup$ Looking at all those quotes, I actually think if you read them with the original definition in mind, they make more sense. But the best examples in Emergence are from Gibbon's Decline and Fall of the Roman Empire, one from Gibbon's personal notes and one from a footnote. The personal note read: "Let us conclude, then, though with some remainder of scepticism, that although Livy's narrative has more of probability, yet that of Polybius has more of truth." And the footnote, to Chapter xxiv of decline and fall, reads: "Such a fact is probable but undoubtable false." $\endgroup$ Mar 20, 2016 at 17:47
-
$\begingroup$ Another really interesting one, also from Emergence, is from a mid 1700s writer named Thomas Church, in response to David Hume's attack on the credibility of miracles... $\endgroup$ Mar 20, 2016 at 17:57
-
$\begingroup$ Quoting from Emergence: "The author is at pains to insist that credibility is relative to the evidence. Church grants, 'that in common discourse, it is not unusual to call any thing credible or incredible, antecedent to our consideration of its proof. But if we examine our ideas, this will be found to be a loose unphilosophical way of expressing ourselves. All that can be meant is, that such a thing is possible or impossible, probable or improbable, or, at farthest, happening very frequently, or very seldom [1750, p. 60]." $\endgroup$ Mar 20, 2016 at 17:57
-
$\begingroup$ But going back to shakespeare quotes, actually, those are pretty fascinating examples. Because most of them do make sense of a sort when defining probable in the modern way, but all of them also make sense if you read them to be referring to how "believable" or "plausible" the subject being referred to is or should be. $\endgroup$ Mar 20, 2016 at 18:05