3
$\begingroup$

If I have a sample $(X_1,Y_1),\dots,(X_n,Y_n)$, is it possible for the sample to be drawn independently, and yet have some of the $X$'s be correlated? If so, I'm trying to understand how a sample can be independent and yet it is possible to have a situation where random variable $X_i$ and $X_j$ be not independent.

Edit:

I was reading this, where the distribution of $X$ is stated as being arbitrary, and the middle of page 2 states that the "samples are all independent". Here no explicit statement was made regarding the correlation between random variables $X_i$ and $X_j$ in different observations in a sample, so I'm unsure if the statement that "samples are all independent" implies that $X_i$ and $X_j$ are independent. In a later lecture in this series (page 1 here), this sample was used in defining the likelihood as $\Pi p(y_i|x_i;b_0,b_1,s^2)$.

From a different source here, at the bottom of page 4 (in the Conditional Likelihood section), the notes state that for data consisting of $(x_i,y_i)$ pairs, "we do not need to assume that the $x_i$ are independent in order to justify the conditional likelihood being a product; we just need to assume that the $y_i$ are independent when each is conditioned on its own $x_i$". The likelihood function defined here has the same form as my previous source.

I'm not sure how the statement in my second source fits in with the first. I'm also not sure if the correlation between $X_i$ and $X_j$ matters (at least in linear regression).

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

To answer this you need to more precise on what you mean by "independence" and "correlation" in this context. In particular, the answer to your question really depends on what you mean when you say that these pairs of values are "drawn independently". It is also unclear what you mean when you ask if some of the $X$s may be correlated, since correlation usually refers either to the underlying true correlation parameter, or the sample correlation for the whole sample.

If the random variables in the draws are statistically independent then they must be uncorrelated (since statistical independence implies zero correlation), and so the true correlation between the values will be zero in this case. Of course, the sample correlation in a finite sample will not usually be zero, so even if you have random variables that are statistically independent, a finite sample of these variables will generally have a sample correlation that is not zero (though for large samples, it will be close).

Unfortunately, it is not really very descriptive to say that values are "drawn independently", since that is not a clear specification of a sampling process. Statistical independence between random variables occurs when they are drawn from an infinitely exchangeable series. This is a reasonable assumption for certain sampling schemes, but it is not really clear here. You have not specified if you are drawing from a finite population, and if so, how? Are you using simple-random sampling with replacement? Without replacement?

Improve this answer
$\endgroup$
8
  • $\begingroup$ I've added more context to my original post. $\endgroup$
    – Yandle
    Commented Apr 15, 2020 at 18:00
  • $\begingroup$ In the linked notes you refer to, yes, they are saying that the pairs of values are mutually independent of one another (so therefore have zero correlation). $\endgroup$
    – Ben
    Commented Apr 15, 2020 at 22:20
  • $\begingroup$ If pairs of RV in each observation is mutually independent of each other, then what's the purpose in stating the assumption that $\epsilon$ is uncorrelated across observations (on the first page here)? In addition, my second source explicitly states that $x$'s need not be independent, does that imply the samples used to construct the conditional likehood is not independent? $\endgroup$
    – Yandle
    Commented Apr 15, 2020 at 22:38
  • $\begingroup$ It is the pairs of observations that are independent of each other. That does not mean that the elements within the pair are independent. That is, they don't mean that $y_i$ is independent of $x_i$, they mean that $(y_i,x_i)$ is independent of $(y_j,x_j)$ for $i \neq j$. $\endgroup$
    – Ben
    Commented Apr 15, 2020 at 22:55
  • $\begingroup$ So in this context, independent samples only mean $p(x_1,y_1,...,x_i,y_i,x_j,y_j,...,x_n,y_n)=p(x_1,y_1)...p(x_i,y_i)p(x_j,y_j)...p(x_n,y_n)$, it is silent on the relationship between $X_i$ and $X_j$ (and between $Y_i$ and $Y_j$)? My main confusion is what the assumed relationship is between $X_i$ and $X_j$ (i.e. correlated, uncorrelated, or independent), and where this would come into play in the context of linear regression. $\endgroup$
    – Yandle
    Commented Apr 15, 2020 at 23:23
0
$\begingroup$

Totally! If you randomly generate independent samples of data over and over, by chance alone you'll get correlated samples.

For instance, try the following R code:

set.seed(123)
corr_list <- c()
for (i in 1:10000) {
  # make some random data
  x <- rnorm(100)
  y <- rnorm(100)
  # calculate correlation
  corr_list[[i]] <- cor(x, y)
 }
hist(corr_list)

enter image description here

Most of the correlations are close to 0, but you can clearly see in the histogram that there are correlations with absolute values > 0.4.

Depending on sample size and the distribution of the population, you can even observe perfect correlations with random data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.