Return to Answer

What i.i.d. assumption states is that random variables are independent and identically distributed. You can formally define what does it mean, but informally it says that all the variables provide the same kind of information independently of each other (you can read also about related exchangeability).

From the abstract ideas let's jump for a moment to concrete example: in most cases your data can be stored in a matrix, with observations row-wise and variables column-wise. If you assume your data to be i.i.d., then it means for you that you need to bother only about relations between columns and do not have to bother about relations between rows. If you bothered about both then you would model dependence of columns on columns and rows on rows, i.e. everything on everything. It is very hard to make simplifications and build a statistical model of everything depending on everything.

Besides your main question you are also asking about cross-validation with non-i.i.d. data. While you seem to understate the importance of i.i.d. assumption, at the same time you overstate the problems of not meeting this assumption poses for cross-validation. There are multiple ways how we can deal with such data when using resampling methods like bootstrap, or cross-validation. If you are dealing with time-series you cannot assume that the values are independent, so taking the random fraction of values would be a bad idea because it would ignore the autocorrelated structure of the data. Because of that, with time-series we commonly use one step ahead cross-validation, i.e. you take part of the series to predict next value (not used for modeling). Similarly, if your data has clustered structure, you sample whole clusters to preserve the nature of the data. So as with modeling, we can deal with non-i.i.d.-sness also when doing cross-validation, but we need to adapt our methods to the nature of the data since methods designed for i.i.d. data do not apply in such cases.

What i.i.d. assumption states is that random variables are independent and identically distributed. You can formally define what does it mean, but informally it says that all the variables provide the same kind of information independently of each other (you can read also about related exchangeability).

From the abstract ideas let's jump for a moment to concrete example: in most cases your data can be stored in a matrix, with observations row-wise and variables column-wise. If you assume your data to be i.i.d., then it means for you that you need to bother only about relations between columns and do not have to bother about relations between rows. If you bothered about both then you would model dependence of columns on columns and rows on rows, i.e. everything on everything. It is very hard to make simplifications and build a statistical model of everything depending on everything.

Besides your main question you are also asking about cross-validation with non-i.i.d. data. While you seem to understate the importance of i.i.d. assumption, at the same time you overstate the problems of not meeting this assumption poses for cross-validation. There are multiple ways how we can deal with such data when using resampling methods like bootstrap, or cross-validation. If you are dealing with time-series you cannot assume that the values are independent, so taking the random fraction of values would be a bad idea because it would ignore the autocorrelated structure of the data. Because of that, with time-series we commonly use one step ahead cross-validation, i.e. you take part of the series to predict next value (not used for modeling). Similarly, if your data has clustered structure, you sample whole clusters to preserve the nature of the data. So as with modeling, we can deal with non-i.i.d.-sness also when doing cross-validation, but we need to adapt our methods to the nature of the data since methods designed for i.i.d. data do not apply in such cases.

Besides your main question you are also asking about cross-validation with non-i.i.d. data. While you seem to understate the importance of i.i.d. assumption, at the same time you overstate the problems that not meeting this assumption poses for cross-validation. There are multiple ways how we can deal with such data when using resampling methods like bootstrap, or cross-validation. If you are dealing with time-series you cannot assume that the values are independent, so taking the random fraction of values would be a bad idea because it would ignore the autocorrelated structure of the data. Because of that, with time-series we commonly use one step ahead cross-validation, i.e. you take part of the series to predict next value (not used for modeling). Similarly, if your data has clustered structure, you sample whole clusters to preserve the nature of the data. So as with modeling, we can deal with non-i.i.d.-sness also when doing cross-validation, but we need to adapt our methods to the nature of the data since methods designed for i.i.d. data do not apply in such cases.

Besides your main question you are also asking about cross-validation with non-i.i.d. data. While you seem to understate the importance of i.i.d. assumption, at the same time you overstate the problems of not meeting this assumption poses for cross-validation. There are multiple ways how we can deal with such data when using resampling methods like bootstrap, or cross-validation. If you are dealing with time-series you cannot assume that the values are independent, so taking the random fraction of values would be a bad idea because it would ignore the autocorrelated structure of the data. Because of that, with time-series we commonly use one step ahead cross-validation, i.e. you take part of the series to predict next value (not used for modeling). Similarly, if your data has clustered structure, you sample whole clusters to preserve the nature of the data. So as with modeling, we can deal with non-i.i.d.-sness also when doing cross-validation, but we need to adapt our methods to the nature of the data since methods designed for i.i.d. data do not apply in such cases.

Imagine that you wanted to learn about abilities of children in a classroom, so you give them some tests. You could use the test results as an indicator of the abilities of kids only if they did them by themselves, independently of each other. If they interacted then you'd probably measure abilities of the most clever kid, or the most influential one. It does not mean that you need to assume that there was no interaction, or dependence, between kids whatsoever, but simply that they did the tests by themselves. The kids also need to be "identically distributed", so they cannot come from different countries, speak different languages, be in different ages since it will make it hard to interpret the results (maybe they did not understand the questions and answered randomly). If you can assume that your data is i.i.d. then you can focus on building a general model. You can deal with non-i.i.d. data but then you have to worry about "noise" in your data much more.

Imagine that you wanted to learn about abilities of children in a classroom, so you give them some tests. You could use the test results as an indicator of the abilities of kids only if they did them by themselves, independently of each other. If they interacted then you'd probably measure abilities of the most clever kid, or the most influential one. It does not mean that you need to assume that there was no interaction, or dependence, between kids whatsoever, but simply that they did the tests by themselves. The kids also need to be "identically distributed", so they cannot come from different countries, speak different languages, be in different ages since it will make it hard to interpret the results (maybe they did not understand the questions and answered randomly). If you can assume that your data is i.i.d. then you can focus on building a general model. You can deal with non-i.i.d. data but then you have to worry about "noise" in your data much more.

Besides your main question you are also asking about cross-validation with non-i.i.d. data. While you seem to understate the importance of i.i.d. assumption, at the same time you overstate the problems that not meeting this assumption poses for cross-validation. There are multiple ways how we can deal with such data when using resampling methods like bootstrap, or cross-validation. If you are dealing with time-series you cannot assume that the values are independent, so taking the random fraction of values would be a bad idea because it would ignore the autocorrelated structure of the data. Because of that, with time-series we commonly use one step ahead cross-validation, i.e. you take part of the series to predict next value (not used for modeling). Similarly, if your data has clustered structure, you sample whole clusters to preserve the nature of the data. So as with modeling, we can deal with non-i.i.d.-sness also when doing cross-validation, but we need to adapt our methods to the nature of the data since methods designed for i.i.d. data do not apply in such cases.