Is independence subjective?

Question

I am trying to better understand exchangeability. Suppose that I would like to run an experiment. I will pick two people and give each a fair coin and tell them to toss N times in a row. The difference is that to the person A I will tell that the coin is fair, and tell person B that I don't know whether the coin is fair or not. But I assure both of them that there is no monkey business going on.

For person A coin tosses are independent and identically distributed, hence they are exchangeable.

For person B coin tosses are neither independent nor identically distributed but they are exchangeable.

What's strange for me at this point is that for the same physical process of tossing fair coins in a row, one person thinks that the tosses are independent and the other doesn't, while I know that they are all IID.

Does this make independence subjective? Or is my understanding incorrect?

Basically B has something to learn about the process while A is unable to learn anything new. Hence their subjective beliefs affects their assessment of independence.

Should we assess independence based on an idealized true data generating process or based on what we currently know about the process?

Here is a similar rhetoric: "Resonance: From Probability To Epistemology And Back"

It seems it depends on whether you are a Bayesian or a Frequentist. Nevertheless it is quite perplexing that if question was posed as randomly picking a coin from an urn containing one biased and one fair coin (not knowing which one you picked) and then tossing the same coin N times, both the frequentist and the bayesian would probably agree that the tosses are not unconditionally independent, which means they both now assess independence based on what they know about the situation instead of the ground truth that the tosses actually don't affect each other.

Answer 1

I'm a bit late and good things have already been written, however I didn't see the following mentioned:

For person A coin tosses are independent and identically distributed, hence they are exchangeable.

For person B coin tosses are neither independent nor identically distributed but they are exchangeable."

This doesn't depend on the information that A or B have, but rather on whether they view probability as epistemic (i.e., referring to knowledge/uncertainty of an agent, like most Bayesians) or as aleatory (referring to data generating processes out there in the world, like most frequentists). One can even be pluralist and accept both views as advantageous in different situations.

In any case, person B can, even with the given information, talk about aleatory probabilities and would then normally model the coin tosses as i.i.d. with unknown probability. It is inappropriate to say that "for person B coin tosses are neither independent nor identically distributed", because the epistemic model in which they are exchangeable but not i.i.d. does not refer to the physical process but rather to the state of knowledge.

What's strange for me at this point is that for the same physical process of tossing fair coins in a row, one person thinks that the tosses are independent and the other doesn't, while I know that they are all IID.

But this is not the case. Person A apparently models what the physical process is, but person B is going on about modelling their own state of knowledge, and does not make statements about the underlying physical process.

Should we assess independence based on an idealized true data generating process or based on what we currently know about the process?

From a pluralist point of view that's your choice and there are pros and cons of both. However once you made your choice, be consistent with it!

Does this make independence subjective? Or is my understanding incorrect?

There are always subjective elements in probability modeling that you cannot get rid of. This is obvious in subjective Bayes, however also frequentists need to make model decisions that are subjective. Probability models are always underdetermined by the data. A frequentist could try to test independence (which doesn't happen very often), but this comes with its own problems and can certainly not detect all conceivable alternative possibilities.

Nevertheless it is quite perplexing that if question was posed as randomly picking a coin from an urn containing one biased and one fair coin (not knowing which one you picked) and then tossing the same coin N times, both the frequentist and the bayesian would probably agree that the tosses are not unconditionally independent, which means they both now assess independence based on what they know about the situation instead of the ground truth that the tosses actually don't affect each other.

If you'd phrase this in terms how they model the situation like this and for what reasons, this would seem far less perplexing than writing about what people think the coin tosses really are. Probability in my view is a tool for modelling the world, not a description of how the world really is.

Answer 2

In probability theory, independence of two events is defined as

$$ P(A \cap B) = P(A)\,P(B) $$

There is nothing subjective about the definition, it is a possible property of random variables. For real life data, because of things like sampling bias, measurement error, insufficient sample size, numerical precision etc., we would be talking about degree to which the assumption of independence was violated. This is one of the reasons why we look at the residual plots of the model.

On one extreme, you can consider if "butterfly’s wings in Brazil set off a tornado in Texas", i.e. say that independence does not exist. Even simple "physical" problems like throwing a coin can possibly be influenced by some external factors (unless it is thrown by a robot at vacuum), yet most people would agree that is is "random" and "independent" enough to not consider such nuances.

On another extreme, you could boldly assume independence, e.g. when building a spam classifier you use Naive Bayes classifier that assumes that all the variables are independent. This would be obviously wrong, as people do not combine words in sentences by pairing them at random. Yet, Naive Bayes classifier can give pretty decent results for many similar problems and is commonly used.

Yes, no matter what assessments people make, they may be influenced by external factors, there is a huge body of research in psychology on this. Independence is one of them, a handy one, that simplifies a lot of maths for solving statistical problems. Usually the problem is not if the assumption is correct or wrong, but on how much would making the wrong assumption influence the results. Moreover, in many cases for computational reason you simply need to make such assumptions, as you wouldn't be able to model every possible interaction of anything with anything in the world.

Answer 3

It seems me that ultimately the answer to your question boil down in the concept of probability that you accept. Let me recover the Tim answer’s

In probability theory, independence of two events is defined as $P(A∩B)=P(A)P(B)$ There is nothing subjective about the definition, it is a possible property of random variables.

The last sentences is strongly questionable in Bruno de Finetti point of view about probability. In the radical interpretation that he suggest, the probabilities themselves are completely subjective, even in case where classical definition is almost always applied (like coin tossing). So independence are subjective judgement also. Note that for exchangeability the same is true.

The experiment that you have in mind is a stylized case for more general question like this: two person face the same probabilistic problem, but one have more information then the other about that.

Therefore the probabilistic model that them keep in mind are almost surely different, and then the conclusions that them achieve.

Let that the conclusions are probability themselves; which is the right one?

In de Finetti point of view the last question are nonsense one. No probability are right because ultimately probability do not exist (in absolutely objective sense). Probabilities (conclusions) can (must) be coherent, no more.

In de Finetti interpretation the conclusions in example can be both right or both wrong or one right and one wrong, regardless of individual knowledge. It depend only from possible logical mistake (inconsistency) in reasoning of both guys.

However is true that in our example the guy B, that have less information, can learn something from the data and if the coin are “truly” interpretable as iid random variables, before or after, should be converge to the conclusion of guy A. Come in place the endless debate between reality/data and probability.

Coming back to your sentence

What's strange for me at this point is that for the same physical process of tossing fair coins in a row, one person thinks that the tosses are independent and the other doesn't, while I know that they are all IID.

Does this make independence subjective?

In de Finetti point of view, the answer is yes and nothing is strange.

N.B: in terminological sense if you "know" that "they are all IID" no other opinion are admissible, then guy B almost surely going to wrong conclusions. However if your "know" is a theoretical assumption the answer to your question ... is in the question (events are independent)... no room for subjective opinions. However note that ask to solve any mathematical problem giving correct terms to some guys and incorrect or incomplete terms to the others is exactly the same. At the other side if your "knowledge" is about reality ... it can be an illusion ... subjective interpretation come back to be admissible.

For full understanding of de Finetti point of view you can read https://www.amazon.com/Theory-Probability-introductory-treatment-Statistics/dp/1119286379

Answer 4

Up to the last paragraph, the setting appears to be entirely Bayesian. I will first respond to the questions raised up to (but excluding) the last paragraph from a Bayesian perspective.

...for the same physical process of tossing fair coins in a row, one person thinks that the tosses are independent and the other doesn't, while I know that they are all IID. Does this make independence subjective?

In your example, the difference between persons arises not because they treat the same information differently, but because of the different (amount of) information they have. The first one is conditioning on $p=0.5$ (using $p$ to denote the probability of heads), while the second one is not. Another, perhaps more relevant question is: Were the two persons given the same information, would they necessarily arrive at the same conclusion regarding independence? If having the same information implies having the same priors and likelihoods (does it? I do not have an answer right now), then the answer is in the affirmative.

Should we assess independence based on an idealized true data generating process or based on what we currently know about the process?

As Bayesians, we should assess independence based on what we currently know about the process. According to Lindley "Understanding Uncertainty" (2006) p. 37-38, Probability describes a relationship between you and the world, or that part of the world involved in the event. <...> It is not solely a feature of your mind, it is not a value possessed by an event but expresses a relationship between you and the event and is a basic tool in your understanding of the world (emphasis is mine). Since independence is defined by probabilities, I think the same logic and interpretation extends to independence.

(It is also not entirely clear to me how to express the statement assess independence based on an idealized true data generating process rigorously in statistical terms.)

---

Regarding the last paragraph and the treatment of independence from a frequentist perspective, there are two steps to the process:

1. How real world events/phehomena are modeled probabilistically
2. How independence is defined and understood within the models (regardless of their connection to the real world).

In step 1, questions can be raised regarding some simplifying assumptions taken when moving from the real world to models. A relevant one for us is about the plausibility of an assumption that the outcome of the first toss does not affect the physical properties of the second toss; this can form a basis for a formal assumption of independence defined on the model level (step 2). In step 2, independence can be assessed formally using laws of probability, and that is relatively easy, I believe (given a complete job in step 1).

Regarding

...it is quite perplexing that if question was posed as randomly picking a coin from an urn containing one biased and one fair coin (not knowing which one you picked) and then tossing the same coin N times, <...> the frequentist <...> would probably agree that the tosses are not unconditionally independent, which means <...> independence [is assessed] based on what they know about the situation instead of the ground truth that the tosses actually don't affect each other,

the problem is with step (1). The real world phenomena are more complicated than just throwing the same coin several times; there is also the action of picking one of two coins in the beginning. Taking proper account of the latter yields (after being formalized in step 2) unconditional dependence. After some thought, one will probably not find it counterintuitive. Now, if we are only looking at tosses of a single coin, then we have independence, whether we define it conditionally on having selected a concrete coin in the experiment you describe or we define it unconditionally in an experiment where only one coin exists (so not your experiment). So in my understanding, there is actually no clash with the ground truth -- as long as we are identifying the phenomena we are modelling / taking into account correctly.

Answer 5

The coin tosses that each of the two subjects will conduct will always be independent and identically distributed random variables (and thus exchangeable) no matter what they think. The underlying properties and distributions are not subject to our beliefs about them.

For instance, if you fill up a room with 1 million people and you give a fair coin to each one and ask them to flip it 20 times chances are that there will be a couple that will get the same side all 20 times. If you go and ask these two specific people what they think about the coin, they will most likely say that they have strong evidence that the coin is biased. However, if you reveal what happened to the rest 999998 people they will take back their previous statements. This example has nothing to do with iid or exchangeability, I just wanted to demonstrate an example when subjective beliefs can be very well structured and scientific and sensible but incorrect compared to what is the "ground truth".

That said, allow me to help you understand the above definitions and concepts through some examples:

Independence: If you flip two coins (don't care if they are fair or biassed) the outcome of the "system" only depends on each of the coin's manufacturing properties and there is no magic wire connecting them and affecting the outcome of one given to what has happened to the other. These two variables are independent. The length of someone's legs and his height are not independent as the outcome "1m tall" has different probabilities if this person's legs are 1.1 meters and 0.5 meters, in the first case the outcome has 0 probability to occur as we are talking about the same person.

Identically distributed: If you have a coin (don't care if its' fair or biased) and you flip it two times the probability of getting heads is the same in the first and in the second flip again because the fact the coin is exactly the same and the first outcome didn't magically destroy any of the coin's underlying properties so that in the second flip one is more likely than the other. If you have two football games: Barcelona vs Real Madrid and Barcelona vs random team in the 3rd league, the probability distributions of the two results are not the same because although the first team is the same in both cases, in the second one is not.

Exchangeable: If you have one coin (don't care if it's fair or biased) and you flip it two times and you write down one outcome in blue and the other outcome in red, then, the probability of the event (blue = heads and red = tails) is the same if the blue is the first flip's outcome and red is the second flip's or the other way round. In other words, the outcome of the coin can give you no clue at all about if it's the first or the second coin flip. The probability of the event (Final score: 7-1) is different if the game is Barcelona vs Random 3rd division team OR if the game is Random 3rd division team vs Barcelona.