0
$\begingroup$

I am working on a problem with a dataset of the form $(x_j,t_i,y_{i,j} )_{(j,i) \in J \times I}$ where $y_{i,j}$ is the output label. Think of progression of illness over time or weather conditions over an area. The labelling is done on new features $x$ that are not in the training set, and no initial $y$ is provided for this $x$.

Here $x_j$ does not vary over time, so the only variable to vary sequentially is time itself. So, I don't see any benefit in using RNN/transformers/LSTM over non-sequential ANN architectures.

In the weather example above I can try to approach this problem by thinking of it as a generalized interpolation $f(x_n,\prod y_{i,j},t_i) = y_{i,n}$, and I can try to generalize this to the illness example. In that case it may make sense for me to use sequential ANN architectures like RNN's. However, I have been specifically asked not to do this.

Obviously, I can try both and pick the one that works the best, but I would still like to know if there's a theoretical construct that can help in picking which approach is the better, or even the most likely better, one.

I have tried to modify the various stochastic versions of well-posedness of problems. However, so far I have not found a satisfactory formulation of an argument despite my intuition saying that it should work; Especially since another part of my intuition says that using an RNN cannot hurt either (it just doesn't confer any benefits over other ANN's)

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

If $y$ is "progression of illness over time or weather conditions" then it does seem to vary over time. The basic forms of time-series models (starting from ARIMA, ending at LSTMs) are univariate models, where the only variable you have is $y$ varying over time. So it sounds like a sequence model. You can use a hybrid model, where there is a sequence component and additional non-varying-over-time features, for example here I mentioned an example of such model where both time-varying and non-varying features were used in a single model.

Improve this answer
$\endgroup$
5
  • $\begingroup$ The reason I am kinda skeptical about using time-series models on these is that if I do that am essentially just transfer learning from the hidden layer for the last feature $x_n$ at time step $t_{final}$ onto the next feature $x_{n+1}$ at time-step $t_1$. This is where I start having issues with this approach. In the illness example, I am transferring hidden features at the end of an illness, i.e. death, onto the start of one. $\endgroup$
    – Avatrin
    Commented May 29, 2021 at 5:47
  • $\begingroup$ @Avatrin I’m not saying you should put the non-sequential features to RNN, you can put them to separate sub-network and combine its output with RNNs sub-network’s output. Also it’s not clear for me what’s your data, “death” in the health data be usually an event ending the series, so it cannot transfer to anything. You seem to have in mind some particular data & model, so maybe instead of asking a general question, ask a separate giving more details about your particular Scranton? $\endgroup$
    – Tim
    Commented May 29, 2021 at 6:20
  • $\begingroup$ When I initially asked the question, I was hoping for a theoretical construct, a criteria, for when to treat a problem as a time-series one, but I guess that doesn't really exist. $\endgroup$
    – Avatrin
    Commented May 29, 2021 at 6:41
  • 1
    $\begingroup$ @Avatin time-series models are for time-series data, if you don’t have such data, you don’t use such models. In your case you have hybrid data, so you need a hybrid model. More detailed answer would depend on what exactly is your data. $\endgroup$
    – Tim
    Commented May 29, 2021 at 7:32
  • $\begingroup$ Well, RNNs are not just for time-series models. RNNs are used to analyze proteins in bioinformatics. Proteins are not time-series data. Heck, they are barely sequential. Their properties can still be predicted using RNNs since they make up long linear chains of amino acids that have different local properties depending upon the amino acids to their right and to their left. So, I still hope for a better criteria for when to use RNNs before I accept an answer. $\endgroup$
    – Avatrin
    Commented May 29, 2021 at 13:09
0
$\begingroup$

The answer to this question is that we are not working with independent and identically distributed random variables.

In most models the samples are assumed to be i.i.d. That is not the case in time-series data, proteins or other sequential data. The inputs and/or the outputs for any given part of the sequence are not independently drawn from the remaining sequence.

We still assume that the different sequences are i.i.d., but the values within each sequence are not. Architectures like RNN's and Transformers try to capture this.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.