81 votes
Accepted

On the importance of the i.i.d. assumption in statistical learning

The i.i.d. assumption about the pairs $(\mathbf{X}_i, y_i)$, $i = 1, \ldots, N$, is often made in statistics and in machine learning. Sometimes for a good reason, sometimes out of convenience and ...
NRH's user avatar
  • 18k
60 votes
Accepted

If X and Y are uncorrelated, are X^2 and Y also uncorrelated?

No. A counterexample: Let $X$ be uniformly distributed on $[-1, 1]$, $Y = X^2$. Then $E[X]=0$ and also $E[XY]=E[X^3]=0$ ($X^3$ is odd function), so $X,Y$ are uncorrelated. But $E[X^2Y] = E[X^4] = E[...
Jakub Bartczuk's user avatar
57 votes
Accepted

Confused about independent probabilities. If a fair coin is flipped 5 times, P(HHHHH) = 0.03125, but P(H | HHHH) = 0.5

P(HHHHH) is the probability of having five heads in a row. But, P(H|HHHH) means having heads if the last four tosses were heads. In the former, you're at the beginning of the experiment and in the ...
gunes's user avatar
  • 57k
49 votes

Example where $X$ and $Z$ are correlated, $Y$ and $Z$ are correlated, but $X$ and $Y$ are independent

Intuitive example: $Z = X + Y$, where $X$ and $Y$ are any two independent random variables with finite nonzero variance.
fblundun's user avatar
  • 3,949
45 votes
Accepted

Does statistical independence mean lack of causation?

So if that's the case, does statistical independence automatically mean lack of causation? No, and here's a simple counter example with a multivariate normal, ...
Carlos Cinelli's user avatar
42 votes

Are two standard normal random variables always independent?

The answer is no. For example, if $X$ is a standard random variable, then $Y=-X$ follows the same statistics, but $X$ and $Y$ are clearly dependent.
zap's user avatar
  • 521
42 votes
Accepted

Why does independence imply zero correlation?

By the definition of the correlation coefficient, if two variables are independent their correlation is zero. So, it couldn't happen to have any correlation by accident! $$\rho_{X,Y}=\frac{\...
OmG's user avatar
  • 1,087
40 votes

Does statistical independence mean lack of causation?

Suppose we have a lightbulb controlled by two switches. Let $S_1$ and $S_2$ denote the state of the switches, which can be either 0 or 1. Let $L$ denote the state of the lighbulb, which can be either ...
user20160's user avatar
  • 32.3k
38 votes

Confused about independent probabilities. If a fair coin is flipped 5 times, P(HHHHH) = 0.03125, but P(H | HHHH) = 0.5

P(HHHHH) There are 32 possible outcomes from flipping a coin 5 times. Here they are listed: ...
Jake Westfall's user avatar
34 votes
Accepted

Zero correlation of all functions of random variables implying independence

Using indicator functions of measurable sets like$$f(x)=\mathbb I_A(x)\quad g(x)=\mathbb I_B(x)$$leads to$$\text{cov}(f(X),g(Y))=\mathbb P(X\in A,Y\in B)-\mathbb P(X\in A)\mathbb P(Y\in B)$$therefore ...
Xi'an's user avatar
  • 104k
30 votes
Accepted

Is "not independent" the same as "dependent" in English?

In statistics, “dependent” and “not independent” have the same meaning. There is no inherent notion of causation. In regular English, I would say that “dependent” implies causation. Dinner temperature ...
Dave's user avatar
  • 61k
28 votes

Simple examples of uncorrelated but not independent $X$ and $Y$

I think the essence of some of the simple counterexamples can be seen by starting with a continuous random variable $X$ centered on zero, i.e. $E[X]=0$. Suppose the pdf of $X$ is even and defined on ...
27 votes

The meaning of "positive dependency" as a condition to use the usual method for FDR control

From your question and in particular your comments to other answers, it seems to me that you are mainly confused about the "big picture" here: namely, what does "positive dependency&...
amoeba's user avatar
  • 104k
26 votes

Are two standard normal random variables always independent?

No, there is no reason to believe that any two standard gaussians are independent. Here's a simple mathematical construction. Suppose that $X$ and $Y$ are two independent standard normal variables. ...
Matthew Drury's user avatar
25 votes
Accepted

Does covariance equal to zero implies independence for binary random variables?

For binary variables their expected value equals the probability that they are equal to one. Therefore, $$ E(XY) = P(XY = 1) = P(X=1 \cap Y=1) \\ E(X) = P(X=1) \\ E(Y) = P(Y=1) \\ $$ If the two ...
gammer's user avatar
  • 1,487
22 votes

A and B are independent. Does P(A ∩ B|C) = P(A|C) · P(B|C) hold?

No this is not in general true, as you can see from a simple counter example: Toss two independent coins. Event $A$ is coin 1 head. $P(A)=0.5$ Event $B$ is coin 2 head. $P(B)=0.5$ Event $C$ is either ...
George Savva's user avatar
  • 2,054
21 votes

Why does independence imply zero correlation?

Comment on sample correlation. In comparing two small independent samples of the same size, the sample correlation is often noticeably different from $r = 0.$ [Nothing here contradicts @OmG's Answer (+...
BruceET's user avatar
  • 55.8k
20 votes

If X and Y are uncorrelated, are X^2 and Y also uncorrelated?

Even if $\operatorname{Corr}(X,Y)=0$, not only is it possible that $X^2$ and $Y$ are correlated, but they may even be perfectly correlated, with $\operatorname{Corr}(X^2,Y)=1$: ...
Silverfish's user avatar
  • 23.2k
20 votes
Accepted

Are linear combinations of independent random variables again independent?

Yes, for the content of your question. and No, for the title, in general. Yes: Your $a_1,\dots, a_n$ are just some constant numbers. Then the independence of $X_1,\dots, X_n$ implies that the $...
Ute's user avatar
  • 2,520
19 votes
Accepted

Proof that joint probability density of independent random variables is equal to the product of marginal densities

By definition, the random variables $X_1,\dots,X_n$ are independent iff $$ \Pr(X_1\in B_1,\dots,X_n\in B_n) = \Pr(X_1\in B_1)\dots\Pr(X_n\in B_n) $$ for every choice of Borel sets $B_1,\dots,B_n$. ...
Zen's user avatar
  • 23.8k
18 votes
Accepted

For intuition, what are some real life examples of uncorrelated but dependent random variables?

In finance, GARCH (generalized autoregressive conditional heteroskedasticity) effects are widely cited here: stock returns $r_t:=(P_t-P_{t-1})/P_{t-1}$, with $P_t$ the price at time $t$, themselves ...
Christoph Hanck's user avatar
18 votes

On the importance of the i.i.d. assumption in statistical learning

What i.i.d. assumption states is that random variables are independent and identically distributed. You can formally define what does it mean, but informally it says that all the variables provide the ...
Tim's user avatar
  • 138k
18 votes
Accepted

Are discrete random variables, with same domain and uniform probability, always independent?

It is simple to construct an example where both variables are marginally uniformly distributed, but they are not independent. The simplest example is to take $X \sim \text{U} \{ -1,0,1 \}$ and let $Y=...
Ben's user avatar
  • 123k
17 votes
Accepted

With categorical data, can there be clusters without the variables being related?

Consider the clear-cluster case with uncorrelated scale variables - such as the top-right picture in the question. And categorize its data. We subdivided the scale range of both variables X and Y ...
ttnphns's user avatar
  • 57.2k
17 votes
Accepted

Independence of sample mean and sample variance in binomial distribution

$\bar x$ and $s^2$ are random variables. We can work out their joint distribution. Let's try the simplest possible nontrivial case, that of a sample of size $2$ from a Binomial$(1,p)$ distribution. ...
whuber's user avatar
  • 321k
17 votes
Accepted

Relation between independence and correlation of uniform random variables

Independent implies uncorrelated but the implication doesn't go the other way. Uncorrelated implies independence only under certain conditions. e.g. if you have a bivariate normal, it is the case that ...
Glen_b's user avatar
  • 281k
17 votes

Let A and B be two random variables, both independent from another random variable C. Is A*B also independent from C?

If all you have is pairwise independence then there is a counterexample. Suppose the following four cases each have probability $\frac14$: ...
Henry's user avatar
  • 38.8k
16 votes
Accepted

Probability of surviving an event three times

When you write "No extra variables, each incident is isolated and does not affect the subsequent", the mathematical word for this is that they are independent. And for independent events $A$ and $B$, ...
Silverfish's user avatar
  • 23.2k
16 votes
Accepted

Under what additional conditions does independence follow from zero correlation?

The statement that you are asking about has two parts: If $X$ and $Y$ are independent, then $X$ and $Y$ are uncorrelated. If $X$ and $Y$ are uncorrelated, then $X$ and $Y$ are independent. Statement ...
Dilip Sarwate's user avatar
16 votes

Why does independence imply zero correlation?

Simple answer: if 2 variables are independent, then the population correlation is zero, whereas the sample correlation will typically be small, but non-zero. That is because the sample is not a ...
Dave's user avatar
  • 507

Only top scored, non community-wiki answers of a minimum length are eligible