From what I understand, Supervised Learning operates under the assumption that the data is I.I.D. It seems to me that the training procedure for RNNs is flawed. We receive observations in a sequential format -- this condradicts the assumption of IID -- that is, observations are temporally dependent on one another. Therefore, it seems to me that RNNs are flawed learning procedures. Am I correct in my reasoning?


Your premise is wrong, it is not true that every (supervised or not) machine learning model assumes that the data consists or independent and identically distributed observations. If it was true, we wouldn't be able to model things like time-series, where the subsequent observations depend on each other, or other cases where the observations are correlated. For such cases we design models that assume dependence and take it into consideration, e.g. the time-series models, or recurrent neural networks. Moreover, you don't need RNN's to find dependence assumption in neural network architectures: in convolutional neural networks you use kernels that share weights and slide through the picture (or something else), so basically you assume that the collection of pixels within the kernel forms some kind of pattern i.e. depends on each other.

You may also find the following thread interesting: On the importance of the i.i.d. assumption in statistical learning

| cite | improve this answer | |
  • $\begingroup$ But I ran into a post by a researcher that said that reported that RNNs were failing for his problem because it assumed the data was IID. $\endgroup$ – coolster1278 Oct 16 '18 at 14:00
  • $\begingroup$ @coolster1278 if you can provide link or exact quote, I could comment on this, but otherwise it would be a guessing game. $\endgroup$ – Tim Oct 16 '18 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.