In statistics, does independent and random describe the same characteristics? What's the difference between them? We often come across the description like "two independent random variables" or "random sampling". I am wondering what's the exact difference between them. Can someone explain this and give some examples? for instance non-independent but random process?
$\begingroup$
$\endgroup$
7
asked Aug 24, 2016 at 8:22
-
$\begingroup$ Here is two distinct (on a not very deep level) concepts merged. "Independent" in the sense independently generated observations, and "independent variables" wrt their distributions. $\endgroup$– ttnphnsCommented Aug 24, 2016 at 14:41
-
3$\begingroup$ This is a strange question, because if you were to consult formal definitions of "random variable" and "independent"--which is what "in statistics" would seem to suggest--you would find they have little in common. $\endgroup$– whuber ♦Commented Aug 24, 2016 at 16:12
-
$\begingroup$ @ttnphns, Yes, I guess I was more confused about the term "independently generated observations" with "randomly generated". In sampling, we often hear (simple) random sampling, which makes me feel like independent samples. I guess if we really want to combine both characteristics in describing a sampling method, it should be: the selection of observations is not dependent on each other (=independently) and the probability of selection an observation is known (=randomly)? $\endgroup$– tiantianchenCommented Aug 24, 2016 at 17:56
-
1$\begingroup$ If we check the definition of independence from wiki: "In probability theory, two events are independent, statistically independent, or stochastically independent if the occurrence of one does not affect the probability of the other.", the dependency of two observations should be based on how they are generated/selected, rather than how they look like in the data. Then the two identical observations in the case I mentioned above should still be independent. $\endgroup$– tiantianchenCommented Aug 24, 2016 at 18:17
-
2$\begingroup$ Please do not confuse the heuristic explanation at the beginning of any Wikipedia entry with a definition. The definition is given under the heading "definition" in the same article. It is the one offered in Tim's answer here. $\endgroup$– whuber ♦Commented Aug 24, 2016 at 18:36
|
Show 2 more comments
7 Answers
$\begingroup$
$\endgroup$
1
I'll try to explain it in non-technical terms: A random variable describes an outcome of an experiment; you can not know in advance what the exact outcome will be but you have some information: you know which outcomes are possible and you know, for each outcome, its probability.
For example, if you toss a fair coin then you do not know in advance whether you will get head or tail, but you know that these are the possible outcomes and you know that each has 50% chance of occurrence.
To explain independence you have to toss two fair coins. After tossing the first coin you know that for the second toss the probabilities of head is still 50% and for tail also. If the first toss has no influence on the probabilities of the second one then both tosses are independent. If the first toss has an influence on the probabilities of the second toss then they are dependent.
An example of dependent tosses is when you glue the two coins together.
-
3$\begingroup$ @immibis Or roll a fair dice, write down the value. then roll it once more and multiply the value with the value written down. This value is random, but dependent on the first roll. $\endgroup$– CrowleyCommented Aug 25, 2016 at 9:44
$\begingroup$
$\endgroup$
9
Random relates to random variable, and independent relates to probabilistic independence. By independence we mean that observing one variable does not tell us anything about the another, or in more formal terms, if $X$ and $Y$ are two random variables, then we say that they are independent if
$$ p_{X,Y}(x, y) = p_X(x)\,p_Y(y) $$
moreover
$$ E(XY) = E(X)E(Y) $$
and their covariance is zero. Random variable $Y$ is dependent on $X$ if it can be written as a function of $X$
$$ Y = f(X) $$
So in this case $Y$ is random and dependent on $X$.
Calling process "non-independent" is pretty misleading - independent of what? I guess you meant that there are some $X_1,\dots,X_k$ independent and identically distributed random variables (check here, or here) that come from some process. By independent we would mean in here that they are independent of each other. There are processes producing dependent random variables, e.g.
$$ X_i = X_{i-1} + \varepsilon $$
where $\varepsilon$ is some random noise. Obviously in such case $X_i$ is dependent on $X_{i-1}$, but it is also random.
-
$\begingroup$ What does $P(X)$ mean if X is a random variable? I think you're confusing RVs and events: two RVs X and Y are independent if the events $P(X\leq r)$ and $P(Y\leq s)$ are independent for all r,s $\endgroup$ Commented Aug 24, 2016 at 19:32
-
$\begingroup$ Then any two continuous random variables are independent. $\endgroup$ Commented Aug 24, 2016 at 19:52
-
$\begingroup$ @m_t_ I really do not think that discussing the notation leads anywhere (see e.g. en.wikipedia.org/wiki/… ) $\endgroup$– TimCommented Aug 24, 2016 at 19:55
-
1$\begingroup$ @m_t_ this is splitting hair, see stats.stackexchange.com/questions/16321/… or math.ucsd.edu/~napkaria/crypto/handouts/IndepDepRV.pdf $\endgroup$– TimCommented Aug 24, 2016 at 20:30
-
2$\begingroup$ @tiantianchen the other way around: if you have iid random variables, then you can construct likelihood function by multiplying the individual pdf's because they are independent. $\endgroup$– TimCommented Aug 24, 2016 at 21:00
$\begingroup$
$\endgroup$
3
The notion of independence is relative, while you can be random by yourself. In your example, you have "two independent random variables", and do not need to talk about several "random sampling".
Suppose you cast a perfect die several times. The outcome $6,5,3,5, 4\ldots$ is a priori random. Knowing the past, you cannot predict the number following 4. Suppose I generate a sequence from the other side of the die: $6\to1$, $3\to4$. I get $1,2,4,2, 3\ldots$. It is as random as the first one. You cannot guess what comes after $3$. But the two sequences are completely dependent.
If one casts two dice in parallel (without interactions between they), their respective sequences will be random and independent.
answered Aug 24, 2016 at 9:11
-
1$\begingroup$ This may be a little technical given the level of the OP, but regarding your statement "You cannot be independent (of something) alone (as a process, a sequence)" consider the following: Any random variable X, which equals a constant c with probability one, is independent of "everything", including itself. I.e., for such an X, X is independent of X. You can easily check that per the definition of independence. $\endgroup$ Commented Aug 24, 2016 at 16:01
-
$\begingroup$ @Mark L. Stone I shall correct this false statement. By alone I meant "in itself". In your definition, are you allowed to say: $X$ is independent, or $X$ and $X$ are independent? $\endgroup$ Commented Aug 24, 2016 at 16:11
-
$\begingroup$ X is independent of itself. I.e., X is independent of X. $\endgroup$ Commented Aug 24, 2016 at 16:13
$\begingroup$
$\endgroup$
2
Variables are used in all fields of mathematics. The definitions for independence and randomness of a variable are applied unilaterally to all forms of mathematics, not just to statistics.
For example, the X and Y axes in 2-dimensional Euclidean geometry represent independent variables, however, their values are not (usually) assigned at random.
Two given variables can be random, or independent (of one another), or both, or neither. Statistics tends to focus on the randomness (more correctly, on probability), and whether or not two variables are independent can have many implications for the probabilities of given outcomes being observed.
You tend to see these two properties (independence and randomness) described together when studying statistics, because both are important to know, and can influence the answer to the question at hand. However, these properties are not synonymous, and in other fields of mathematics they do not necessarily occur together.
-
$\begingroup$ Thanks. Can you explain more about "whether two variables are independent can have many implications for the probabilities of given outcomes being observed." $\endgroup$ Commented Aug 24, 2016 at 20:22
-
3$\begingroup$ This is a non-statistical answer that addresses a different sense of "independent" than the one used in the question. It also confuses two senses of "variable": one is the mathematical one and the other is the statistical definition of random variable (which definitely is not the same as variables on geometric axes). $\endgroup$– whuber ♦Commented Aug 24, 2016 at 20:28
$\begingroup$
$\endgroup$
1
When you have a pair of values when the first is randomly generated and the second has any dependence on the first one. e.g. height and weight of a man. There is correlation between them. But they are both random.
-
$\begingroup$ Although this post uses the words "random" and "dependent," it doesn't define them or clearly distinguish them. Indeed, it seems to suggest that "random=dependent"! $\endgroup$– whuber ♦Commented Sep 8, 2016 at 17:47
$\begingroup$
$\endgroup$
3
The coin example is a great illustration of a random and independent variable, a good good way to think of a random but dependent variable would be the next card drawn from a seven deck shoe of playing cards, the -likelihood- of any specific numerical outcome changes depending on the cards previously dealt, but until only one value of card remains in the shoe, the value of the card to come next will remain random.
-
3$\begingroup$ Probably worth replacing the word "likelihood" by "probability" here, since likelihood has a separate technical definition in statistics $\endgroup$ Commented Aug 24, 2016 at 23:04
-
1$\begingroup$ A probability that depends on other events (often previous events, but sometimes based on knowledge of future or simultaneous events - there actually is no temporal direction to this) is called a conditional probability. The word likelihood is used to refer to a kind of "probability in reverse" (or in the continuous case, a probability density) - that is, one calculates the probability of an outcome (e.g. your data) conditional on your model parameter(s), but if we think of this the other way round, it's the likelihood of that parameter, given your data. $\endgroup$ Commented Sep 30, 2016 at 10:51
-
1$\begingroup$ Unless you're calculating the likelihood of a parameter, it's best to avoid the word "likelihood" in statistics, even where in normal English one would use "likelihood" as a synonym for the probability of an event (e.g. "rolling ten sixes in a row in a game of dice has a very low likelihood" is fine for colloquial English, but isn't using the word correctly in a statistical sense). "Let $π$ be a parameter denoting the probability that a biased die rolls a six; calculate the likelihood that $π=1/6$ given that ten rolls of the die were all sixes" is statistically correct but jargony $\endgroup$ Commented Sep 30, 2016 at 11:43
$\begingroup$
$\endgroup$
20
David Bohm in his work Causality and Chance in Modern Physics (London: Routledge, 1957/1984) describes causality, chance, randomness, and independence:
"In nature nothing remains constant. Everything is in a perpetual state of transformation, motion, and change. However, we discover that nothing simply surges up out of nothing without having antecedents that existed before. Likewise, nothing ever disappears without a trace, in the sense that it gives rise to absolutley nothing existing at later times. .... everything comes from other things and gives rise to other things. This principle is not yet a statement of the existence of causality in nature. To come to causality, the next step is then to note that as we study processes taking place under a wide range of conditions, we discover that inside all of the complexity of change and transformation there are relationships that remain effectively constant. .... At this point, however, we meet a new problem. For the necessity of a causal law is never absolute. Thus, we see that one must conceive of the law of nature as necessary only if one abstracts from contingencies, representing essentially independent factors which may exist outside the scope of things that can be treated by the laws under consideration, and which do not follow necessarily from anything that may be specified under the context of these laws. Such contingencies lead to chance." (pp.1-2)
"The tendency for contingencies lying outside a given context to fluctuate independently of happenings inside that context has demonstrated itself to be so widespread that one may enunciate it as a principle; namely, the principle of randomness. By randomness we mean just that this independence leads to fluctuation of these contingencies in a very complicated way over a wide range of possibliities, but in such a manner that statistical averages have a regular and approximately predictable behaviour." (p.22)
-
3$\begingroup$ Your definition of "random" seems unusual. It appears to be intimately connected with concepts of "predictability" and "pattern"--but what exactly do those mean? For instance, if an experiment that potentially could yield any number between $0$ and $1$ were consistently observed to yield values of either $1/3$ or $4/7$, that would seem to be a "pattern" and--to the extent it differs from the original infinite set of possible values--is at least partially "predictable." Where you write "if you plot..." it seems you are claiming that no univariate variable can be random! $\endgroup$– whuber ♦Commented Aug 24, 2016 at 16:10
-
3$\begingroup$ You seem to be discussing stochastic processes (in time) rather than randomness and random variables. $\endgroup$– whuber ♦Commented Aug 24, 2016 at 16:31
-
4$\begingroup$ I believe part of the difficulty we are having in communicating is that you appear to be thinking of "independent" in the sense of an independent variable in regression. Although some elements of the question might suggest that, the phrases "two independent random variables" and "random sampling" indicate otherwise. $\endgroup$– whuber ♦Commented Aug 24, 2016 at 19:12
-
1$\begingroup$ I cannot even tell what your understanding is, because your answer provides no definitions. I'm having to guess what you're trying to say from the examples and descriptions you give. They appear to differ from the senses of "random" and "independent" in the ways I have described in previous comments. $\endgroup$– whuber ♦Commented Aug 24, 2016 at 20:33
-
1$\begingroup$ I'd add to @whuber comments that your definition mentioning random variables influencing each other may be misleading. "Influence" is a very strong term implicating some kind of causality etc. while the formal definition of independence does not require any causality or influence but it is simply about relations of joint vs individual probabilities. $\endgroup$– TimCommented Aug 24, 2016 at 21:38