2
$\begingroup$

I am reading the paper "An Introduction to variable and feature selection"

by Guyon et al., and got stumbled about the following (page 1163):

(1) They construct two datasets by drawing for each of the two variables at random 100 examples from a 2-dimensional Gaussian distribution with standard deviation 1.

(2) The class centers are placed at coordinates (-1;-1) and (1;1)

(3) There is zero covariance:

Two classes with gaussian distributions each, histograms for projections on the axis. The scatter plot is also shown with axis swap

Now they state the following: "(..) in spite of the fact that the examples are i.i.d. with respect to the class conditional distributions, the variables are correlated because of the separation of the class center positions" - I am not quite sure I understand this statement (would it mean that if we randomly sample from any distribution and then shift one of the datasets by adding some constant, the datasets are correlated?)

Thanks

$\endgroup$

2 Answers 2

2
$\begingroup$

I am not sure I fully understand your question but here is what, in my opinion, is a reasonable interpretation.

Let $X$ and $Y$ denote the two random variables. Then, the graphics seem to indicate that conditioned on the class, $X$ and $Y$ are conditionally independent normal random variables with identical means and variances (which we shall take to be $1$ for convenience). Thus, conditioned on Class $A^+$, $$f_{X,Y\mid A^+}(x,y\mid A^+) = \frac{1}{2\pi}\exp\left[-\left.\left.\frac 12 \right((x-1)^2 + (y-1)^2\right)\right]\tag{1}$$ while conditioned on Class $A^-$, $$f_{X,Y\mid A^+}(x,y\mid A^-) = \frac{1}{2\pi}\exp\left[-\left.\left.\frac 12 \right((x+1)^2 + (y+1)^2\right)\right]\tag{2}$$ $X$ and $Y$ are conditionally independent, and hence conditionally uncorrelated given the class. However, the unconditional joint density of $X$ and $Y$ is the mixture density \begin{align} f_{X,Y}(x,y) &= f_{X,Y\mid A^+}(x,y\mid A^+)P(A^+) + f_{X,Y\mid A^-}(x,y\mid A^-)P(A^-)\\ &= \frac{P(A^+)}{2\pi}\exp\left[-\left.\left.\frac 12 \right((x-1)^2 + (y-1)^2\right)\right] + \frac{P(A^-)}{2\pi}\exp\left[-\left.\left.\frac 12 \right((x+1)^2 + (y+1)^2\right)\right] \end{align} which is not a bivariate normal density and is not the product of the unconditional marginal densities of $X$ and $Y$ which marginal densities are not themselves normal densities (they are a mixture of normal densities).

So, if $X$ and $Y$ are not unconditionally independent, are they nonetheless unconditionally uncorrelated? Well,

$$E[X] = E[X\mid A^+]P(A^+)+E[X\mid A^-]P(A^-) = P(A^+)-P(A^-)$$ and similarly $E[Y]= P(A^+)-P(A^-)$. Similarly, we have $$E[XY]=E[XY\mid A^+]P(A^+) + E[XY\mid A^-)P(A^-) = P(A^+)+P(A^-) = 1$$ since $E[XY\mid A^+]=E[X\mid A^+]E[Y\mid A^+] = (+1)^2$ while $E[XY\mid A^-] = (-1)^2$. Thus,

$$\operatorname{cov}(X,Y)=E[XY]-E[X]E[Y] = 1 - (P(A^+-P(A^-))^2 > 0$$

except in the trivial case when one of $P(A^+)$ and $P(A^-)$ has value $1$ (and the other has value $0$), that is, we really have only one class.

Edit in response to OP's further questions

If you are given the entire data set $\{(x_i,y_i)\}$ of black and white points but are not told which points are black and which are white, then the two coordinates are correlated. If you regress $y$ on $x$, you should get something very close to $\hat{y} = x$, that is, given the $x$ coordinate of a point is $x_j$, the best linear unbiased estimate of its $y$ coordinate is $x_j$. I say "very close to" because you are making this estimate based on the data points and so using estimated values for the means, variances, covariances, etc. In the probability model described above, $\hat{y} = x$ is an exact result.

If you are given the entire data set $\{(x_i,y_i)\}$ but are told which points are black and which are white, then regressing $y$ on $x$ with the black points should get something very close to $\hat{y} = \bar{y}_{\text{black}}$, that is, given the $x$ coordinate of a black point is $x_j$, the best estimate of its $y$ coordinate is $\bar{y}_{\text{black}}$, the average $y$ coordinate of the black points. Note that effectively, you can ignore the $x$ coordinate in estimating the $y$ coordinate of a black point because the two are uncorrelated (actually independent). I say "very close to" because you are making this estimate based on the data points and so using estimated values for the means, variances, covariances, etc. In the probability model described above, $\hat{y} = +1$ is an exact result. Similarly, for white points mutatis mutandis with $\hat{y} = -1$ being an exact result.

If you are given a point $(x_j,y_j)$ and asked "Is this a black point or a white point?", the answer in the probability model described above is to reply "Black" or "White" according as $x_j+y_j$ is greater than $0$ or less than $0$. For fence-sitters with $x_j+y_j$ exactly equal to $0$, toss a fair coin. I will let you choose whether to say "Black" or "White" when the coin turns up Heads (and, of course, the opposite when the coin turns up Tails).

$\endgroup$
1
  • $\begingroup$ thanks I think I understand your explanation - however if I understand it correctly it is not exactly the same statement as the responder above referring to coordinates is making..? $\endgroup$
    – Pugl
    Oct 15, 2015 at 10:18
4
$\begingroup$

It means that the $X$ and $Y$ coordinates are correlated. The correlation is of the form: if $X$ is low, it's more likely to be the lower-left cluster, so $Y$ is likely to be low.

See also Simpson's paradox.

$\endgroup$
3
  • $\begingroup$ In short because of the shifted centers, we have this dependence of coordinates and hence also not UNconditional independence, right? $\endgroup$
    – Pugl
    Oct 7, 2015 at 20:12
  • $\begingroup$ Yes, it's because the $X$ and $Y$ distributions look different in each cluster, so by knowing $X$, you have some information about the cluster and thus also about $Y$: in other words, a correlation. Note that if the clusters were offset horizontally, not diagonally, there would be no correlation: knowing $X$ would tell you the cluster, but the cluster would tell you nothing about $Y$. $\endgroup$
    – Creosote
    Oct 7, 2015 at 21:36
  • $\begingroup$ But is this really the statement that is done in the quote? I understood when they talk about "the variables are correlated", they are referring to a correlation between the variables of the different datasets (and not to the coordinates of a variable of the same dataset). And so, even in the case where the clusters are only separated horizontally, and we do not gain any information by knowing any one coordinate, do we not have a correlation between variables of the different sets? $\endgroup$
    – Pugl
    Oct 8, 2015 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.