6
$\begingroup$

I am trying to better understand exchangeability. Suppose that I would like to run an experiment. I will pick two people and give each a fair coin and tell them to toss N times in a row. The difference is that to the person A I will tell that the coin is fair, and tell person B that I don't know whether the coin is fair or not. But I assure both of them that there is no monkey business going on.

For person A coin tosses are independent and identically distributed, hence they are exchangeable.

For person B coin tosses are neither independent nor identically distributed but they are exchangeable.

What's strange for me at this point is that for the same physical process of tossing fair coins in a row, one person thinks that the tosses are independent and the other doesn't, while I know that they are all IID.

Does this make independence subjective? Or is my understanding incorrect?

Basically B has something to learn about the process while A is unable to learn anything new. Hence their subjective beliefs affects their assessment of independence.

Should we assess independence based on an idealized true data generating process or based on what we currently know about the process?

Here is a similar rhetoric: "Resonance: From Probability To Epistemology And Back"

It seems it depends on whether you are a Bayesian or a Frequentist. Nevertheless it is quite perplexing that if question was posed as randomly picking a coin from an urn containing one biased and one fair coin (not knowing which one you picked) and then tossing the same coin N times, both the frequentist and the bayesian would probably agree that the tosses are not unconditionally independent, which means they both now assess independence based on what they know about the situation instead of the ground truth that the tosses actually don't affect each other.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ How exactly telling someone that the tosses are independent makes them so? If you told a person that they are tossing a frog instead of a coin, would it make a frog out of a coin? $\endgroup$
    – Tim ♦
    Feb 1 '20 at 8:46
  • 1
    $\begingroup$ Even if B's tosses may be biased with an unknown probability, they are still likely to be independent given that probability $\endgroup$
    – Henry
    Feb 1 '20 at 8:49
  • $\begingroup$ @Tim I didn’t tell them anything about independence. Basically their assessment of independence is affected by what they know about the coin. $\endgroup$
    – Cagdas Ozgenc
    Feb 1 '20 at 10:05
  • $\begingroup$ @Henry yes they are conditionally independent. But that’s not my point. $\endgroup$
    – Cagdas Ozgenc
    Feb 1 '20 at 10:05
  • $\begingroup$ @RichardHardy If we are clear about the definition of independence, then write down the entire model and then derive unconditional distribution of the toss. You will see that they are not independent under the definition we agreed on. stats.stackexchange.com/a/312503/20980 $\endgroup$
    – Cagdas Ozgenc
    Feb 9 '20 at 18:56
4
$\begingroup$

I'm a bit late and good things have already been written, however I didn't see the following mentioned:

For person A coin tosses are independent and identically distributed, hence they are exchangeable.

For person B coin tosses are neither independent nor identically distributed but they are exchangeable."

This doesn't depend on the information that A or B have, but rather on whether they view probability as epistemic (i.e., referring to knowledge/uncertainty of an agent, like most Bayesians) or as aleatory (referring to data generating processes out there in the world, like most frequentists). One can even be pluralist and accept both views as advantageous in different situations.

In any case, person B can, even with the given information, talk about aleatory probabilities and would then normally model the coin tosses as i.i.d. with unknown probability. It is inappropriate to say that "for person B coin tosses are neither independent nor identically distributed", because the epistemic model in which they are exchangeable but not i.i.d. does not refer to the physical process but rather to the state of knowledge.

What's strange for me at this point is that for the same physical process of tossing fair coins in a row, one person thinks that the tosses are independent and the other doesn't, while I know that they are all IID.

But this is not the case. Person A apparently models what the physical process is, but person B is going on about modelling their own state of knowledge, and does not make statements about the underlying physical process.

Should we assess independence based on an idealized true data generating process or based on what we currently know about the process?

From a pluralist point of view that's your choice and there are pros and cons of both. However once you made your choice, be consistent with it!

Does this make independence subjective? Or is my understanding incorrect?

There are always subjective elements in probability modeling that you cannot get rid of. This is obvious in subjective Bayes, however also frequentists need to make model decisions that are subjective. Probability models are always underdetermined by the data. A frequentist could try to test independence (which doesn't happen very often), but this comes with its own problems and can certainly not detect all conceivable alternative possibilities.

Nevertheless it is quite perplexing that if question was posed as randomly picking a coin from an urn containing one biased and one fair coin (not knowing which one you picked) and then tossing the same coin N times, both the frequentist and the bayesian would probably agree that the tosses are not unconditionally independent, which means they both now assess independence based on what they know about the situation instead of the ground truth that the tosses actually don't affect each other.

If you'd phrase this in terms how they model the situation like this and for what reasons, this would seem far less perplexing than writing about what people think the coin tosses really are. Probability in my view is a tool for modelling the world, not a description of how the world really is.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ "Person A apparently models what the physical process is" This may not be true. The problem is Person A is given no choice. He is given p=1/2. From this point on you may claim that he is modelling his belief as well as he is modelling the physical process. You cannot distinguish the two. $\endgroup$
    – Cagdas Ozgenc
    Feb 11 '20 at 15:04
  • $\begingroup$ You're right that person A can use this model for both the physical process or their state of information, and person A has to decide what they mean. As you made person A up, that's for you to decide (that person A models the physical process was just my interpretation of your writing). Still what I wrote stands: as long as person B uses epistemic probability (which is the standard thing for an exchangeability non-iid model), B makes no statement about whether the physical process is "really" independent. $\endgroup$
    – Christian Hennig
    Feb 11 '20 at 15:10
  • $\begingroup$ If you looked at the question (except for the last paragraph) from a Bayesian perspective, I think statements starting with It is inappropriate to say and But this is not the case would not apply since the problems they address would not actually be problems. Other than that, I am pleased to see that our answers broadly agree on the points that are covered in both of them. $\endgroup$
    – Richard Hardy
    Feb 11 '20 at 20:01
  • $\begingroup$ @Richard Hardy: It seems we only disagree about our interpretation of the question wording, which is somewhat ambiguous. They refer to the physical process, therefore I thought I shouldn't interpret it all as Bayesian. $\endgroup$
    – Christian Hennig
    Feb 11 '20 at 21:58
2
$\begingroup$

In probability theory, independence of two events is defined as

$$ P(A \cap B) = P(A)\,P(B) $$

There is nothing subjective about the definition, it is a possible property of random variables. For real life data, because of things like sampling bias, measurement error, insufficient sample size, numerical precision etc., we would be talking about degree to which the assumption of independence was violated. This is one of the reasons why we look at the residual plots of the model.

On one extreme, you can consider if "butterfly’s wings in Brazil set off a tornado in Texas", i.e. say that independence does not exist. Even simple "physical" problems like throwing a coin can possibly be influenced by some external factors (unless it is thrown by a robot at vacuum), yet most people would agree that is is "random" and "independent" enough to not consider such nuances.

On another extreme, you could boldly assume independence, e.g. when building a spam classifier you use Naive Bayes classifier that assumes that all the variables are independent. This would be obviously wrong, as people do not combine words in sentences by pairing them at random. Yet, Naive Bayes classifier can give pretty decent results for many similar problems and is commonly used.

Yes, no matter what assessments people make, they may be influenced by external factors, there is a huge body of research in psychology on this. Independence is one of them, a handy one, that simplifies a lot of maths for solving statistical problems. Usually the problem is not if the assumption is correct or wrong, but on how much would making the wrong assumption influence the results. Moreover, in many cases for computational reason you simply need to make such assumptions, as you wouldn't be able to model every possible interaction of anything with anything in the world.

Improve this answer
$\endgroup$
4
  • $\begingroup$ So you think that In the case that I don’t know the bias of the coin previous toss result doesn’t affect the distribution of the next toss? Previous outcomes doesn’t make me get smarter about the coin simply by counting how many heads came previously? $\endgroup$
    – Cagdas Ozgenc
    Feb 1 '20 at 14:13
  • 2
    $\begingroup$ @CagdasOzgenc I didn't say anything like this. $\endgroup$
    – Tim ♦
    Feb 1 '20 at 15:28
  • $\begingroup$ From a Bayesian perspective (which I believe the OP is using in the entire post except for the last paragraph), the statement independence ... it is a possible property of random variables is probably incorrect; see my answer, especially the quote from Lindley (2006). See also the answer by @markowitz. $\endgroup$
    – Richard Hardy
    Feb 11 '20 at 12:31
  • 1
    $\begingroup$ Also, your answer focuses on how real world events are modeled probabilistically and evaluating some simplifying assumptions encountered in the process. Another perspective would be to look at probabilistic abstractions (random variables, models) by themselves, excluding the real world, and see what can be said about the subjectivity of independence within them. This is not intended as any kind of criticism, just as a thought that I arrived at while trying to understand the essence of the question and the answers. $\endgroup$
    – Richard Hardy
    Feb 11 '20 at 12:32
2
$\begingroup$

It seems me that ultimately the answer to your question boil down in the concept of probability that you accept. Let me recover the Tim answer’s

In probability theory, independence of two events is defined as $P(A∩B)=P(A)P(B)$ There is nothing subjective about the definition, it is a possible property of random variables.

The last sentences is strongly questionable in Bruno de Finetti point of view about probability. In the radical interpretation that he suggest, the probabilities themselves are completely subjective, even in case where classical definition is almost always applied (like coin tossing). So independence are subjective judgement also. Note that for exchangeability the same is true.

The experiment that you have in mind is a stylized case for more general question like this: two person face the same probabilistic problem, but one have more information then the other about that.

Therefore the probabilistic model that them keep in mind are almost surely different, and then the conclusions that them achieve.

Let that the conclusions are probability themselves; which is the right one?

In de Finetti point of view the last question are nonsense one. No probability are right because ultimately probability do not exist (in absolutely objective sense). Probabilities (conclusions) can (must) be coherent, no more.

In de Finetti interpretation the conclusions in example can be both right or both wrong or one right and one wrong, regardless of individual knowledge. It depend only from possible logical mistake (inconsistency) in reasoning of both guys.

However is true that in our example the guy B, that have less information, can learn something from the data and if the coin are “truly” interpretable as iid random variables, before or after, should be converge to the conclusion of guy A. Come in place the endless debate between reality/data and probability.

Coming back to your sentence

What's strange for me at this point is that for the same physical process of tossing fair coins in a row, one person thinks that the tosses are independent and the other doesn't, while I know that they are all IID.

Does this make independence subjective?

In de Finetti point of view, the answer is yes and nothing is strange.

N.B: in terminological sense if you "know" that "they are all IID" no other opinion are admissible, then guy B almost surely going to wrong conclusions. However if your "know" is a theoretical assumption the answer to your question ... is in the question (events are independent)... no room for subjective opinions. However note that ask to solve any mathematical problem giving correct terms to some guys and incorrect or incomplete terms to the others is exactly the same. At the other side if your "knowledge" is about reality ... it can be an illusion ... subjective interpretation come back to be admissible.

For full understanding of de Finetti point of view you can read https://www.amazon.com/Theory-Probability-introductory-treatment-Statistics/dp/1119286379

Improve this answer
$\endgroup$
2
  • $\begingroup$ Very nice summary. Is there an extension to probability theory to distinguish our beliefs from the convergence of those beliefs once we observe infinite data? For example subject B knows that the coin is not tempered with at every toss. He also knows that he is learning from the process. So tosses are not independent under the current information set, but independent once he receives the necessary piece. He can somehow foresee this because my understanding of definetti is that at end he needs to converge to an IID sequence from a mixture of IID sequence. So something like independent in limit. $\endgroup$
    – Cagdas Ozgenc
    Feb 2 '20 at 19:38
  • $\begingroup$ About this point I remember only one phrase where de Finetti warns that if we face an infinite sequence, then the independence hypothesis can be convenient but is stronger assumption then seems to be. $\endgroup$
    – markowitz
    Feb 2 '20 at 21:32
1
$\begingroup$

Up to the last paragraph, the setting appears to be entirely Bayesian. I will first respond to the questions raised up to (but excluding) the last paragraph from a Bayesian perspective.

...for the same physical process of tossing fair coins in a row, one person thinks that the tosses are independent and the other doesn't, while I know that they are all IID. Does this make independence subjective?

In your example, the difference between persons arises not because they treat the same information differently, but because of the different (amount of) information they have. The first one is conditioning on $p=0.5$ (using $p$ to denote the probability of heads), while the second one is not. Another, perhaps more relevant question is: Were the two persons given the same information, would they necessarily arrive at the same conclusion regarding independence? If having the same information implies having the same priors and likelihoods (does it? I do not have an answer right now), then the answer is in the affirmative.

Should we assess independence based on an idealized true data generating process or based on what we currently know about the process?

As Bayesians, we should assess independence based on what we currently know about the process. According to Lindley "Understanding Uncertainty" (2006) p. 37-38, Probability describes a relationship between you and the world, or that part of the world involved in the event. <...> It is not solely a feature of your mind, it is not a value possessed by an event but expresses a relationship between you and the event and is a basic tool in your understanding of the world (emphasis is mine). Since independence is defined by probabilities, I think the same logic and interpretation extends to independence.

(It is also not entirely clear to me how to express the statement assess independence based on an idealized true data generating process rigorously in statistical terms.)

Regarding the last paragraph and the treatment of independence from a frequentist perspective, there are two steps to the process:

  1. How real world events/phehomena are modeled probabilistically
  2. How independence is defined and understood within the models (regardless of their connection to the real world).

In step 1, questions can be raised regarding some simplifying assumptions taken when moving from the real world to models. A relevant one for us is about the plausibility of an assumption that the outcome of the first toss does not affect the physical properties of the second toss; this can form a basis for a formal assumption of independence defined on the model level (step 2). In step 2, independence can be assessed formally using laws of probability, and that is relatively easy, I believe (given a complete job in step 1).

Regarding

...it is quite perplexing that if question was posed as randomly picking a coin from an urn containing one biased and one fair coin (not knowing which one you picked) and then tossing the same coin N times, <...> the frequentist <...> would probably agree that the tosses are not unconditionally independent, which means <...> independence [is assessed] based on what they know about the situation instead of the ground truth that the tosses actually don't affect each other,

the problem is with step (1). The real world phenomena are more complicated than just throwing the same coin several times; there is also the action of picking one of two coins in the beginning. Taking proper account of the latter yields (after being formalized in step 2) unconditional dependence. After some thought, one will probably not find it counterintuitive. Now, if we are only looking at tosses of a single coin, then we have independence, whether we define it conditionally on having selected a concrete coin in the experiment you describe or we define it unconditionally in an experiment where only one coin exists (so not your experiment). So in my understanding, there is actually no clash with the ground truth -- as long as we are identifying the phenomena we are modelling / taking into account correctly.

Improve this answer
$\endgroup$
8
  • $\begingroup$ In general there is not a big fuss. even the de Finetti's theorem concludes that we are dealing with a mixture of iid sequences, which is comforting. However it is still problematic from the perspective that independence prohibits improvement on predictive capability, i.e. $p(T_1,T_0) = p(T_1)p(T_0) => p(T_1|T_0) = p(T_1)$. Text in your answer doesn't account for this gap. Freqentist probably will have to stop and reiterate his iid assumption at each time step (this time it is iid, this time it is iid, I swear this time it is iid). $\endgroup$
    – Cagdas Ozgenc
    Feb 11 '20 at 13:55
  • $\begingroup$ @CagdasOzgenc, in which perspective, Bayesian or frequentist, does independence prohibit improvement on predictive ability? A frequentist would use $\hat{p}$, not $p$ for constructing predictions, and the estimation precision of $\hat{p}$ benefits from each additional observation of a throw of a given coin. A Bayesian would use $p$ for prediction, but there is dependence in the Bayesian perspective, so independence is not a problem there. Did you mean anything concrete by Text in your answer? I also did not quite get your last sentence on why a frequentist would reiterate the assumption. $\endgroup$
    – Richard Hardy
    Feb 11 '20 at 14:06
  • $\begingroup$ Why would frequentist use $\hat{p}$ for prediction (rhetorical)? He starts with the notion that his estimate is useless for future, meaning that he can also use the unconditional distribution of the next toss without using any information up until that point. So s/he looks in conflict with him/herself. As I said it is not a big fuss. But still looks a little bit fishy to me. $\endgroup$
    – Cagdas Ozgenc
    Feb 11 '20 at 14:30
  • $\begingroup$ @CagdasOzgenc, why would a frequentist ever start from the notion that his/her estimate is useless for the future? Is independence of observations supposed to imply that? Almost all frequentist tools are built on the assumption of (conditionally or unconditionally) independent observations. We build models, estimate parameters and based on the estimated models we do prediction. $\endgroup$
    – Richard Hardy
    Feb 11 '20 at 14:37
  • $\begingroup$ The way frequentist recovers from $p(T_1|T_0) = p(T_1)$ is by adding identical assumption. philsci-archive.pitt.edu/12059/2/DeFinettiTheo.pdf Remark 1 $\endgroup$
    – Cagdas Ozgenc
    Feb 11 '20 at 14:46
0
$\begingroup$

The coin tosses that each of the two subjects will conduct will always be independent and identically distributed random variables (and thus exchangeable) no matter what they think. The underlying properties and distributions are not subject to our beliefs about them.

For instance, if you fill up a room with 1 million people and you give a fair coin to each one and ask them to flip it 20 times chances are that there will be a couple that will get the same side all 20 times. If you go and ask these two specific people what they think about the coin, they will most likely say that they have strong evidence that the coin is biased. However, if you reveal what happened to the rest 999998 people they will take back their previous statements. This example has nothing to do with iid or exchangeability, I just wanted to demonstrate an example when subjective beliefs can be very well structured and scientific and sensible but incorrect compared to what is the "ground truth".

That said, allow me to help you understand the above definitions and concepts through some examples:

Independence: If you flip two coins (don't care if they are fair or biassed) the outcome of the "system" only depends on each of the coin's manufacturing properties and there is no magic wire connecting them and affecting the outcome of one given to what has happened to the other. These two variables are independent. The length of someone's legs and his height are not independent as the outcome "1m tall" has different probabilities if this person's legs are 1.1 meters and 0.5 meters, in the first case the outcome has 0 probability to occur as we are talking about the same person.

Identically distributed: If you have a coin (don't care if its' fair or biased) and you flip it two times the probability of getting heads is the same in the first and in the second flip again because the fact the coin is exactly the same and the first outcome didn't magically destroy any of the coin's underlying properties so that in the second flip one is more likely than the other. If you have two football games: Barcelona vs Real Madrid and Barcelona vs random team in the 3rd league, the probability distributions of the two results are not the same because although the first team is the same in both cases, in the second one is not.

Exchangeable: If you have one coin (don't care if it's fair or biased) and you flip it two times and you write down one outcome in blue and the other outcome in red, then, the probability of the event (blue = heads and red = tails) is the same if the blue is the first flip's outcome and red is the second flip's or the other way round. In other words, the outcome of the coin can give you no clue at all about if it's the first or the second coin flip. The probability of the event (Final score: 7-1) is different if the game is Barcelona vs Random 3rd division team OR if the game is Random 3rd division team vs Barcelona.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ It is not about whether coin is biased or not. It is about the fact that you don’t know how much bias there is. That’s why previous tosses give you information about the next toss. $\endgroup$
    – Cagdas Ozgenc
    Feb 1 '20 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.