2
$\begingroup$

From what I understand, Supervised Learning operates under the assumption that the data is I.I.D. It seems to me that the training procedure for RNNs is flawed. We receive observations in a sequential format -- this condradicts the assumption of IID -- that is, observations are temporally dependent on one another. Therefore, it seems to me that RNNs are flawed learning procedures. Am I correct in my reasoning?

$\endgroup$

1 Answer 1

4
$\begingroup$

Your premise is wrong, it is not true that every (supervised or not) machine learning model assumes that the data consists or independent and identically distributed observations. If it was true, we wouldn't be able to model things like time-series, where the subsequent observations depend on each other, or other cases where the observations are correlated. For such cases we design models that assume dependence and take it into consideration, e.g. the time-series models, or recurrent neural networks. Moreover, you don't need RNN's to find dependence assumption in neural network architectures: in convolutional neural networks you use kernels that share weights and slide through the picture (or something else), so basically you assume that the collection of pixels within the kernel forms some kind of pattern i.e. depends on each other.

You may also find the following thread interesting: On the importance of the i.i.d. assumption in statistical learning

$\endgroup$
2
  • $\begingroup$ But I ran into a post by a researcher that said that reported that RNNs were failing for his problem because it assumed the data was IID. $\endgroup$ Oct 16, 2018 at 14:00
  • $\begingroup$ @coolster1278 if you can provide link or exact quote, I could comment on this, but otherwise it would be a guessing game. $\endgroup$
    – Tim
    Oct 16, 2018 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.