Correlation despite Independence for Random Variables I am reading the paper "An Introduction to variable and feature selection"
by Guyon et al., and got stumbled about the following (page 1163):
(1) They construct two datasets by drawing for each of the two variables at random 100 examples from a 2-dimensional Gaussian distribution with standard deviation 1.
(2) The class centers are placed at coordinates (-1;-1) and (1;1)
(3) There is zero covariance:

Now they state the following: "(..) in spite of the fact that the examples are i.i.d. with respect to the class conditional distributions, the variables are correlated because of the separation of the class center positions" - I am not quite sure I understand this statement (would it mean that if we randomly sample from any distribution and then shift one of the datasets by adding some constant, the datasets are correlated?)
Thanks
 A: It means that the $X$ and $Y$ coordinates are correlated.  The correlation is of the form: if $X$ is low, it's more likely to be the lower-left cluster, so $Y$ is likely to be low.
See also Simpson's paradox.
A: I am not sure I fully understand your question but here is what,
in my opinion, is a reasonable interpretation.
Let $X$ and $Y$ denote the two random variables. Then, the graphics
seem to indicate that conditioned on the class, $X$ and $Y$
are conditionally independent normal random variables
with identical means and variances (which we shall take to be $1$
for convenience). Thus, conditioned on Class $A^+$, 
$$f_{X,Y\mid A^+}(x,y\mid A^+) = \frac{1}{2\pi}\exp\left[-\left.\left.\frac 12
\right((x-1)^2 + (y-1)^2\right)\right]\tag{1}$$
while 
conditioned on Class $A^-$, 
$$f_{X,Y\mid A^+}(x,y\mid A^-) = \frac{1}{2\pi}\exp\left[-\left.\left.\frac 12
\right((x+1)^2 + (y+1)^2\right)\right]\tag{2}$$
$X$ and $Y$ are conditionally independent, and hence
conditionally uncorrelated given the class. However, the
unconditional joint density of $X$ and $Y$ is the
mixture density
\begin{align}
f_{X,Y}(x,y) &= f_{X,Y\mid A^+}(x,y\mid A^+)P(A^+)
+ f_{X,Y\mid A^-}(x,y\mid A^-)P(A^-)\\
&= \frac{P(A^+)}{2\pi}\exp\left[-\left.\left.\frac 12
\right((x-1)^2 + (y-1)^2\right)\right]
+ \frac{P(A^-)}{2\pi}\exp\left[-\left.\left.\frac 12
\right((x+1)^2 + (y+1)^2\right)\right]
\end{align}
which is not a bivariate normal density and is not the
product of the unconditional marginal densities of $X$ and $Y$
which marginal densities are not themselves
normal densities (they are a mixture of normal densities).
So, if $X$ and $Y$ are not unconditionally independent, are they
nonetheless unconditionally uncorrelated?  Well, 
$$E[X] = E[X\mid A^+]P(A^+)+E[X\mid A^-]P(A^-) = P(A^+)-P(A^-)$$
and similarly $E[Y]= P(A^+)-P(A^-)$. Similarly, we have
$$E[XY]=E[XY\mid A^+]P(A^+) + E[XY\mid A^-)P(A^-)
= P(A^+)+P(A^-) = 1$$
since $E[XY\mid A^+]=E[X\mid A^+]E[Y\mid A^+] = (+1)^2$
while $E[XY\mid A^-] = (-1)^2$. Thus,

$$\operatorname{cov}(X,Y)=E[XY]-E[X]E[Y] = 1 - (P(A^+-P(A^-))^2 > 0$$

except in the trivial case
when one of $P(A^+)$ and $P(A^-)$ has value $1$
(and the other has value $0$), that is, we really have only one
class.
Edit in response to OP's further questions
If you are given the entire data set $\{(x_i,y_i)\}$
of black and white points but are not told which points are black and which are white, then the two
coordinates are correlated. If you regress $y$ on $x$, you should
get something very close to $\hat{y} = x$, that is, given the $x$
coordinate of a point is $x_j$, the best linear unbiased
estimate of its $y$ coordinate is $x_j$. I say "very close to" because you are making this estimate
based on the data points and so using estimated values for the
means, variances, covariances, etc. In the probability model 
described above, $\hat{y} = x$ is an exact result.  
If you are given the entire data set $\{(x_i,y_i)\}$ but
are told which points are black and which are white, then
regressing $y$ on $x$ with the black points should get something very close to $\hat{y} = \bar{y}_{\text{black}}$, that is, given the $x$
coordinate of a black point is $x_j$, the best estimate of its $y$ coordinate is $\bar{y}_{\text{black}}$, the average $y$ coordinate of the 
black points. Note that effectively, you can ignore the $x$
coordinate in estimating the $y$ coordinate of a black point
because the two
are uncorrelated (actually independent).
I say "very close to" because you are making this estimate
based on the data points and so using estimated values for the
means, variances, covariances, etc. In the probability model 
described above, $\hat{y} = +1$ is an exact result. Similarly, for
white points mutatis mutandis with $\hat{y} = -1$ being an exact
result.
If you are given a point $(x_j,y_j)$ and asked "Is this a black
point or a white point?", the answer in the probability model described
above is to reply "Black" or "White" according as $x_j+y_j$ is
greater than $0$ or less than $0$. For fence-sitters with
$x_j+y_j$ exactly equal to $0$, toss a fair coin. I will let you
choose whether to say "Black" or "White" when the coin turns up
Heads (and, of course, the opposite when the coin turns up Tails).
