What [*i.i.d.* assumption][1] states is that random variables are **[independent][2]** and **[identically distributed][3]**. You can formally define what does it mean, but informally it says that all the variables provide the *same kind of information* independently of each other (you can read also about related [exchangeability][4]). From the abstract ideas let's jump for a moment to concrete example: in most cases your data can be stored in a matrix, with observations row-wise and variables column-wise. If you assume your data to be *i.i.d.*, then it means for you that you need to bother only about relations between columns and do not have to bother about relations between rows. If you bothered about both then you would model dependence of columns on columns and rows on rows, i.e. everything on everything. It is very hard to make simplifications and build a statistical [model][5] of everything depending on everything. You correctly noticed that exchengeability makes it possible for us to use methods such as cross-validation, or bootstrap, but it also makes it possible to use [central limit theorem][6] and it enables us to make simplifications helpful for modeling (thinking in column-wise terms). As you noticed in the LASSO example, independence assumption is often softened to [conditional independence][7]. Even in such case we need independent and identically distributed "parts". Similar, softer assumption is often made for time-series models, that you mentioned, that assume [stationarity][8] (so there is dependence but there is also a common distribution and series stabilizes over time -- again "i.i.d." parts). It is a matter of observing a number of similar things that carry the same idea about some general phenomenon. If we have a number of distinct and dependent things we cannot make any generalizations. What you have to remember is that this is *only* an assumption, we are not strict about it. It is about having enough things that all, independently, convey similar information about some common phenomenon. If the things influenced each other they would obviously convey similar information so they wouldn't be that useful. Imagine that you wanted to learn about abilities of children in a classroom, so you give them some tests. You could use the test results as an indicator of the abilities of kids only if they did them by themselves, independently of each other. If they interacted then you'd probably measure abilities of the most clever kid, or the most influential one. It does not mean that you need to assume that there was no interaction, or dependence, between kids whatsoever, but simply that they did the tests by themselves. The kids also need to be "identically distributed", so they cannot come from different countries, speak different languages, be in different ages since it will make it hard to interpret the results (maybe they did not understand the questions and answered randomly). If you can assume that your data is i.i.d. then you can focus on building a general model. You can deal with non-i.i.d. data but then you have to worry about "noise" in your data much more. [1]: http://stats.stackexchange.com/questions/82096/is-independent-and-identically-distributed-an-assumption-or-a-fact [2]: https://en.wikipedia.org/wiki/Independence_%28probability_theory%29 [3]: https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables#cite_note-1 [4]: https://en.wikipedia.org/wiki/Exchangeable_random_variables [5]: http://stats.stackexchange.com/questions/210403/in-laymans-terms-what-is-the-difference-between-a-model-and-a-distribution/210419#210419 [6]: https://en.wikipedia.org/wiki/Central_limit_theorem [7]: https://en.wikipedia.org/wiki/Conditional_independence#Uses_in_Bayesian_inference [8]: https://en.wikipedia.org/wiki/Stationary_process