1
$\begingroup$

I have seen two posts about this before (Are RNNs inherently flawed? Supervised Learning assumes IID data but sequential data is not IID, Realistically, does the i.i.d. assumption hold for the vast majority of supervised learning tasks?), however, the answers did not really satisfy me - thus I would like to open this topic again.

I am wondering how the iid assumption about our data distribution works for recurrent problems / models. Say we apply an LSTM to a sequential problem. We usually put up an iid assumption, because we use MLE to find best parameters, i.e. maximize P(X|θ), with X being the data and θ the parameters, and for calculating this break this down to P(x_1|θ) * ... * P(x_n|θ) (more specifically, assume we are doine classification, thus "data" and the x_i's are actually our predictions).

Now assume our first task is classifying full sequences, i.e. having multiple points per sequence and only outputting one prediction per sequence. Then, I believe the iid assumption still correctly holds. However, when we now predict one label in each step, these depend on previous predictions, and our above made assumption and reformulation of the likelihood should be wrong - is that correct?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Time series don't have i.i.d. components, otherwise there's no need to model them sequentially taking time into account but we could treat each sample individually (shuffle all of them and learn distribution). In general we assume that the distribution of a given sample depends on the past samples as you said. But this doesn't contradicts the assumption that each of the sequences (not samples of each sequence) in the dataset are i.i.d.

Say you have a speech dataset. You can consider each chunk or each audio as a vector and each of such vectors to be i.i.d (not amplitude values but a representation of the speech sequence).

To illustrate this better, you can think of the problem of modelling images. You can vectorize the image or treat it as a matrix. You usually have a lot of statistical dependence between neighboring pixels (that's what image coders exploit) but still you can consider each image to be i.i.d (each vector or each matrix come from the same distribution).

Hope this helps you clarify.

PS: For the formulation of the joint probability of a sequence and how to predict sequences with CNNs you can check the famous WaveNet paper.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for your answer! I understand your points, still, unfortunately am not sure about one thing: Lets say we input speech and output words (roughly assuming there is a one to one mapping between "one" speech input and the written word, so in every step there is one input and one output). I understand, the different sequences are iid, so that does not matter here. However, when trying such a model I would maximize the likelihood of the given data, namely sth like P(Y|θ), where Y are my predictions for that sequence given my parameters $\endgroup$
    – user143877
    Commented Apr 16, 2020 at 10:49
  • $\begingroup$ (could e.g. modelled by cross-entropy to the ground truth or whatever). Then, what applying the cross-entropy function actually does imo is writing this as P(y_1|θ) * ... * P(y_n|θ), e.g. calculating cross-entropy between every predicted work and every target word, and multiplying, and then averaging for example. Here, as far as I see it, we still make use of the iid assumption between the data points (otherwise we could not decompose our likelihood as such) - isn't that, theoretically a problem? I hope, I explained my problem somewhat clearly ... $\endgroup$
    – user143877
    Commented Apr 16, 2020 at 10:49
  • $\begingroup$ If I get what you say, even if you are evaluating elementwise differences with your target, it doesn't mean you need this i.i.d. assumption. For example, when fitting a line in a scatter plot it makes perfect sense to use MSE to assess your model even if you are modelling some correlation, right?. Using cross-entropy (as WaveNet does) is more of the same, but instead you do not assume any distribution as MSE does (if you penalize near errors less as in MSE you implicitely assume some shape of error distribution). You can assume the error is independent but not the data itself. $\endgroup$
    – Oriol B
    Commented Apr 16, 2020 at 11:13
  • $\begingroup$ Thats very helpful, thanks - just to clarify one last time: youre saying, it is important that our predictions roughly are iid, which basically is the case for sequential models, not that the data is? further, for RL this yields problems because there we are interested in sth like that because there our "predictions" are actions or value estimations, and these are not iid because the agent actually is taking actions, reaching different states? $\endgroup$
    – user143877
    Commented Apr 16, 2020 at 11:20
  • $\begingroup$ What I am saying is that if the model is "good enough", then the error is independent. If you see classical models for time series such as ARIMA, one way to know if the model fits the data correctly is to do statistical tests on the error signal to see if it corresponds to noise (usually white Gaussian). So note that the predictions and the data are not independent. And if noise has some time dependence, you can augment the capacity of the model, stack another LSTM... then it makes sense to assume independence. $\endgroup$
    – Oriol B
    Commented Apr 16, 2020 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.