In statistics, does independent and random describe the same characteristics? What's the difference between them? We often come across the description like "two independent random variables" or "random sampling". I am wondering what's the exact difference between them. Can someone explain this and give some examples? for instance non-independent but random process?

  • Here is two distinct (on a not very deep level) concepts merged. "Independent" in the sense independently generated observations, and "independent variables" wrt their distributions. – ttnphns Aug 24 '16 at 14:41
  • 3
    This is a strange question, because if you were to consult formal definitions of "random variable" and "independent"--which is what "in statistics" would seem to suggest--you would find they have little in common. – whuber Aug 24 '16 at 16:12
  • @ttnphns, Yes, I guess I was more confused about the term "independently generated observations" with "randomly generated". In sampling, we often hear (simple) random sampling, which makes me feel like independent samples. I guess if we really want to combine both characteristics in describing a sampling method, it should be: the selection of observations is not dependent on each other (=independently) and the probability of selection an observation is known (=randomly)? – tiantianchen Aug 24 '16 at 17:56
  • 1
    If we check the definition of independence from wiki: "In probability theory, two events are independent, statistically independent, or stochastically independent if the occurrence of one does not affect the probability of the other.", the dependency of two observations should be based on how they are generated/selected, rather than how they look like in the data. Then the two identical observations in the case I mentioned above should still be independent. – tiantianchen Aug 24 '16 at 18:17
  • 2
    Please do not confuse the heuristic explanation at the beginning of any Wikipedia entry with a definition. The definition is given under the heading "definition" in the same article. It is the one offered in Tim's answer here. – whuber Aug 24 '16 at 18:36
up vote 34 down vote accepted

I'll try to explain it in non-technical terms: A random variable describes an outcome of an experiment; you can not know in advance what the exact outcome will be but you have some information: you know which outcomes are possible and you know, for each outcome, its probability.

For example, if you toss a fair coin then you do not know in advance whether you will get head or tail, but you know that these are the possible outcomes and you know that each has 50% chance of occurrence.

To explain independence you have to toss two fair coins. After tossing the first coin you know that for the second toss the probabilities of head is still 50% and for tail also. If the first toss has no influence on the probabilities of the second one then both tosses are independent. If the first toss has an influence on the probabilities of the second toss then they are dependent.

An example of dependent tosses is when you glue the two coins together.

  • 3
    Another pair of dependent variables would be "whether you got heads" and "whether you got tails". Both are random but they are not independent of each other. – immibis Aug 25 '16 at 0:05
  • 3
    @immibis Or roll a fair dice, write down the value. then roll it once more and multiply the value with the value written down. This value is random, but dependent on the first roll. – Crowley Aug 25 '16 at 9:44

Random relates to random variable, and independent relates to probabilistic independence. By independence we mean that observing one variable does not tell us anything about the another, or in more formal terms, if $X$ and $Y$ are two random variables, then we say that they are independent if

$$ p_{X,Y}(x, y) = p_X(x)\,p_Y(y) $$

moreover

$$ E(XY) = E(X)E(Y) $$

and their covariance is zero. Random variable $Y$ is dependent on $X$ if it can be written as a function of $X$

$$ Y = f(X) $$

So in this case $Y$ is random and dependent on $X$.

Calling process "non-independent" is pretty misleading - independent of what? I guess you meant that there are some $X_1,\dots,X_k$ independent and identically distributed random variables (check here, or here) that come from some process. By independent we would mean in here that they are independent of each other. There are processes producing dependent random variables, e.g.

$$ X_i = X_{i-1} + \varepsilon $$

where $\varepsilon$ is some random noise. Obviously in such case $X_i$ is dependent on $X_{i-1}$, but it is also random.

  • What does $P(X)$ mean if X is a random variable? I think you're confusing RVs and events: two RVs X and Y are independent if the events $P(X\leq r)$ and $P(Y\leq s)$ are independent for all r,s – Matthew Towers Aug 24 '16 at 19:32
  • Then any two continuous random variables are independent. – Matthew Towers Aug 24 '16 at 19:52
  • @m_t_ I really do not think that discussing the notation leads anywhere (see e.g. en.wikipedia.org/wiki/… ) – Tim Aug 24 '16 at 19:55
  • 1
  • 2
    @tiantianchen the other way around: if you have iid random variables, then you can construct likelihood function by multiplying the individual pdf's because they are independent. – Tim Aug 24 '16 at 21:00

Variables are used in all fields of mathematics. The definitions for independence and randomness of a variable are applied unilaterally to all forms of mathematics, not just to statistics.

For example, the X and Y axes in 2-dimensional Euclidean geometry represent independent variables, however, their values are not (usually) assigned at random.

Two given variables can be random, or independent (of one another), or both, or neither. Statistics tends to focus on the randomness (more correctly, on probability), and whether or not two variables are independent can have many implications for the probabilities of given outcomes being observed.

You tend to see these two properties (independence and randomness) described together when studying statistics, because both are important to know, and can influence the answer to the question at hand. However, these properties are not synonymous, and in other fields of mathematics they do not necessarily occur together.

  • Thanks. Can you explain more about "whether two variables are independent can have many implications for the probabilities of given outcomes being observed." – tiantianchen Aug 24 '16 at 20:22
  • 3
    This is a non-statistical answer that addresses a different sense of "independent" than the one used in the question. It also confuses two senses of "variable": one is the mathematical one and the other is the statistical definition of random variable (which definitely is not the same as variables on geometric axes). – whuber Aug 24 '16 at 20:28

The notion of independence is relative, while you can be random by yourself. In your example, you have "two independent random variables", and do not need to talk about several "random sampling".

Suppose you cast a perfect die several times. The outcome $6,5,3,5, 4\ldots$ is a priori random. Knowing the past, you cannot predict the number following 4. Suppose I generate a sequence from the other side of the die: $6\to1$, $3\to4$. I get $1,2,4,2, 3\ldots$. It is as random as the first one. You cannot guess what comes after $3$. But the two sequences are completely dependent.

If one casts two dice in parallel (without interactions between they), their respective sequences will be random and independent.

  • 1
    This may be a little technical given the level of the OP, but regarding your statement "You cannot be independent (of something) alone (as a process, a sequence)" consider the following: Any random variable X, which equals a constant c with probability one, is independent of "everything", including itself. I.e., for such an X, X is independent of X. You can easily check that per the definition of independence. – Mark L. Stone Aug 24 '16 at 16:01
  • @Mark L. Stone I shall correct this false statement. By alone I meant "in itself". In your definition, are you allowed to say: $X$ is independent, or $X$ and $X$ are independent? – Laurent Duval Aug 24 '16 at 16:11
  • X is independent of itself. I.e., X is independent of X. – Mark L. Stone Aug 24 '16 at 16:13

When you have a pair of values when the first is randomly generated and the second has any dependence on the first one. e.g. height and weight of a man. There is correlation between them. But they are both random.

  • Although this post uses the words "random" and "dependent," it doesn't define them or clearly distinguish them. Indeed, it seems to suggest that "random=dependent"! – whuber Sep 8 '16 at 17:47

The coin example is a great illustration of a random and independent variable, a good good way to think of a random but dependent variable would be the next card drawn from a seven deck shoe of playing cards, the -likelihood- of any specific numerical outcome changes depending on the cards previously dealt, but until only one value of card remains in the shoe, the value of the card to come next will remain random.

  • 3
    Probably worth replacing the word "likelihood" by "probability" here, since likelihood has a separate technical definition in statistics – Silverfish Aug 24 '16 at 23:04
  • 1
    A probability that depends on other events (often previous events, but sometimes based on knowledge of future or simultaneous events - there actually is no temporal direction to this) is called a conditional probability. The word likelihood is used to refer to a kind of "probability in reverse" (or in the continuous case, a probability density) - that is, one calculates the probability of an outcome (e.g. your data) conditional on your model parameter(s), but if we think of this the other way round, it's the likelihood of that parameter, given your data. – Silverfish Sep 30 '16 at 10:51
  • 1
    Unless you're calculating the likelihood of a parameter, it's best to avoid the word "likelihood" in statistics, even where in normal English one would use "likelihood" as a synonym for the probability of an event (e.g. "rolling ten sixes in a row in a game of dice has a very low likelihood" is fine for colloquial English, but isn't using the word correctly in a statistical sense). "Let $π$ be a parameter denoting the probability that a biased die rolls a six; calculate the likelihood that $π=1/6$ given that ten rolls of the die were all sixes" is statistically correct but jargony – Silverfish Sep 30 '16 at 11:43

David Bohm in his work Causality and Chance in Modern Physics (London: Routledge, 1957/1984) describes causality, chance, randomness, and independence:

"In nature nothing remains constant. Everything is in a perpetual state of transformation, motion, and change. However, we discover that nothing simply surges up out of nothing without having antecedents that existed before. Likewise, nothing ever disappears without a trace, in the sense that it gives rise to absolutley nothing existing at later times. .... everything comes from other things and gives rise to other things. This principle is not yet a statement of the existence of causality in nature. To come to causality, the next step is then to note that as we study processes taking place under a wide range of conditions, we discover that inside all of the complexity of change and transformation there are relationships that remain effectively constant. .... At this point, however, we meet a new problem. For the necessity of a causal law is never absolute. Thus, we see that one must conceive of the law of nature as necessary only if one abstracts from contingencies, representing essentially independent factors which may exist outside the scope of things that can be treated by the laws under consideration, and which do not follow necessarily from anything that may be specified under the context of these laws. Such contingencies lead to chance." (pp.1-2)

"The tendency for contingencies lying outside a given context to fluctuate independently of happenings inside that context has demonstrated itself to be so widespread that one may enunciate it as a principle; namely, the principle of randomness. By randomness we mean just that this independence leads to fluctuation of these contingencies in a very complicated way over a wide range of possibliities, but in such a manner that statistical averages have a regular and approximately predictable behaviour." (p.22)

  • 3
    Your definition of "random" seems unusual. It appears to be intimately connected with concepts of "predictability" and "pattern"--but what exactly do those mean? For instance, if an experiment that potentially could yield any number between $0$ and $1$ were consistently observed to yield values of either $1/3$ or $4/7$, that would seem to be a "pattern" and--to the extent it differs from the original infinite set of possible values--is at least partially "predictable." Where you write "if you plot..." it seems you are claiming that no univariate variable can be random! – whuber Aug 24 '16 at 16:10
  • 3
    You seem to be discussing stochastic processes (in time) rather than randomness and random variables. – whuber Aug 24 '16 at 16:31
  • 4
    I believe part of the difficulty we are having in communicating is that you appear to be thinking of "independent" in the sense of an independent variable in regression. Although some elements of the question might suggest that, the phrases "two independent random variables" and "random sampling" indicate otherwise. – whuber Aug 24 '16 at 19:12
  • 1
    I cannot even tell what your understanding is, because your answer provides no definitions. I'm having to guess what you're trying to say from the examples and descriptions you give. They appear to differ from the senses of "random" and "independent" in the ways I have described in previous comments. – whuber Aug 24 '16 at 20:33
  • 1
    I'd add to @whuber comments that your definition mentioning random variables influencing each other may be misleading. "Influence" is a very strong term implicating some kind of causality etc. while the formal definition of independence does not require any causality or influence but it is simply about relations of joint vs individual probabilities. – Tim Aug 24 '16 at 21:38

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.