Timeline for On the importance of the i.i.d. assumption in statistical learning
Current License: CC BY-SA 3.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 24, 2016 at 9:55 | comment | added | Quantuple | Yes absolutely... I wrote too fast and it resulted in a very unclear comment. When I wrote "distribution of input data $x$ is of no concern to the modeller", I was actually thinking about the fact that the independent part of the iid assumption is not relevant when estimating a model (since it doesn't impact the regression function $E[y \vert X]$). As far as the identical part of the iid assumption is concerned, it is indeed a necessary assumption to set the whole statistical inference wheel into motion (it avoids in your words that "every data point is generated by different mechanism"). | |
| May 24, 2016 at 9:37 | comment | added | mpiktas | Now pick a training set $i=1,...,n/2$ and a test set $i=n/2+1,...,n$. No matter which training method you chose, it will perform horribly on the test set, since the data is generated by two different processes, which are not identical. This is a contrived example, but nothing precludes it from happening in real statistical learning example. | |
| May 24, 2016 at 9:34 | comment | added | mpiktas | Distribution of input data is important. If we do not assume that the distribution of data is somehow fixed, then we can have no confidence that training will result in a robust model, i.e. one that perform well on test data. Suppose that iid assumption fails, or rather that independence assumption is not violated but the data need not be identicaly distributed. This means that DGP can be the following: $y_i = \alpha + \beta_1 x_{1i} + \varepsilon_i$ for $i=1,...,n/2$ and $y_i=\alpha+\beta_2x_{2i}+\varepsilon_i$, for $i=n/2+1,...,n$. Suppose $x_{1i}$ and $x_{2i}$ are independent. | |
| May 24, 2016 at 9:04 | comment | added | Quantuple | (ctd) ... but as you've stated it in your first bullet point, the assumption of iid training examples will come back when we'll be looking at the generalisation properties of the LASSO. What would be nice (and what I am desperately looking for I guess) is a reference/simple technical explanation which shows how the violation of the iid assumption introduces an optimistic bias in the cross-validation estimator for instance. | |
| May 24, 2016 at 9:02 | comment | added | Quantuple | Thanks your interesting take on the question. As far as your first point is concerned, it is indeed easy to conceive that the iid assumption will spring in somewhere in the reasoning, but would you happen to have a reference (not that I don't believe, just that I would like to know where exactly). Your second point is crystal clear and I had never thought of it that way. But for training, this distribution of "input" data $x$ is of no concern to the modeller in general, right? In the LASSO example, we are only concerned in the conditional independent of responses $y$ given inputs $x$ (ctd) ... | |
| May 24, 2016 at 7:18 | history | answered | mpiktas | CC BY-SA 3.0 |