importance of independence among random variables

Question

I always read random variables as being independent and identically distributed. I understand the concept of being identically distributed, because if different random variables are distributed in more ways than one, then getting a pdf of a combination of random variables would be extremely difficult.

But what about 'independence'? Why do we always say: Let $X_1, X_2, X_3, \cdots, X_n$ be iid?

What would happen if $X_1, X_2, X_3, \cdots, X_n$ are NOT independent but identically distributed?

Your insights would be great.

Answer 1

iid is generally used to describe a sample $S=\{x_1,\dots,x_n\}$ of random variables $X_1,\dots,X_n$. Being independent refers to how the (physical) process of collecting the sample was performed and it assures the representative fairness of the sampling. Dependent samples introduce bias into the results, e.g. if on a voting-intention study both, husband and wife, were asked about their voting intention, we would get, high likely, dependent answers; thus, miss-representing the population.

From a computational point of view, independency significantly simplifies operations due to the factorization $P(X_1\leq x_1,\dots,X_n\leq x_n)=P(X_1\leq x_1)\cdots P(X_n\leq x_n)$, avoiding a possibly curse on dimensionality. For instance, consider the painful computation of a maximum-likelihood estimator based on a dependent sample.

However, it is not always the case that we are interested in iid samples. Times series analysis is based on the dependency among the random variables, as many other stochastic processes are. Dependence and conditional dependence structures are at the core of models such as markov chain and, more generally, graphical models.

Answer 2

$X_1, X_2, X_3, \cdots, X_n$

That's okay if you say non-identical distribution are difficult to combine, linear or product, but you can't combine them easily if RVs are dependent.

Independence of RVs is more important than identical, you can combine independent and non-identical distribution see. But you can't combine easily for non-independent and identical distributions.

as convolution demand for independent RVs, and

Joint probability distribution for independent RVs is just product of RVs, otherwise you need to calculate from start.you can find Joint probability distribution for 2 dependent RVs. But if you go for more non-independent RVs you shall follow Chain Rule which will be getting difficult.

Answer 3

Primarily, if they are not independent, the complexity of the problem explodes. Admittedly, the assumption of iid is sometimes poor and the complexity is then justified.

Indeed, if you take treat time of discovery as a measure of complexity, typical linear regression has been around for ages, and only works properly if there is independence between Y and the error term. Linear regression is ancient, and highly versatile otherwise. If Y is correlated with itself, say over time, then you have autocorrelation, and it becomes substantially harder and the development and maturation of time series is far far later.

Answer 4

What would happen if $X_1, X_2, X_3, \cdots, X_n$ are NOT independent but identically distributed?

Nothing would happen if the variables are not independent but are identically distributed; it is what will not happen that is of interest. You will not be able to use several results that need the random variables to be independent.

In fact, non-independent but identically distributed random variables occur more commonly than you think. For example, consider an urn with $n$ balls numbered from $1$ to $n$. The experiment consists of drawing $k \leq n$ balls from the urn. The random variable $X_i$ is the number on the $i$-th ball drawn.

• If the balls are drawn with replacement, the $X_i$ are independent identically distributed random variables; each $X_i$ is a discrete random variable uniformly distributed on $\{1,2,\ldots, n\}$. Note that $X_i$ and $X_j$ can have the same value (same ball was drawn both times).

• If the balls are drawn without replacement, the $X_i$ are dependent identically distributed random variables; however, as before, each $X_i$ is a discrete random variable uniformly distributed on $\{1,2,\ldots, n\}$ but $X_i$ and $X_j$ cannot have the same value: they are dependent random variables.