4
$\begingroup$

I'm wondering whether anyone had success in including continuous variables in an LSTM model?

For example, assume that each output should have a prediction for amount of apples and amount of oranges found:

Simple example:

x1 = [3, "apples", 5, "oranges"]
y1 = [3, 5]

More complex:

x1 = ["I", "want", "3", "apples", "and", 5, "oranges", "now"]
y1 = [3, 5]

x2 = ["No", "apples", 1, "orange"]
y2 = [0, 1]

x3 = [1000, "oranges", "and", 11, "apples"]
y3 = [11, 1000]

x4 = [10, "times", 10, "oranges"]
y4 = [0, 100]

It is not viable to treat numbers as categorical variables in this case. I would even more like if it is able to learn multiplication, like in the last case.

So, did anyone find a sequence model that can incorporate continuous variables in the inputs (mixed with textual)?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ You should spell out LSTM. What does that mean? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Sep 16, 2016 at 12:45
  • $\begingroup$ @kjetilbhalvorsen My apologies. LSTM is not explained in the first thing you google? Has my google been trained on me? Anyways, it stands for Long Short Term Memory (RNN) networks. Oh and RNN stands for Recurrent Neural Network :) $\endgroup$
    – PascalVKooten
    Commented Sep 21, 2016 at 18:54

1 Answer 1

Reset to default
3
$\begingroup$

So, did anyone find a sequence model that can incorporate continuous variables?

Yes. You can use continuous variables as input (e.g. when using word embeddings, example) or as output (e.g., regression, example)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Nice papers, but unfortunately I still do not see "mixed" data containing textual sequence + continuous variable in there? The second paper works with only numeric data. I'm looking at this from a perspective of a lot of textual data, with every now and then a numeric value. We unfortunately cannot really have a token to represent each number :( $\endgroup$
    – PascalVKooten
    Commented Sep 21, 2016 at 19:01
  • $\begingroup$ @PascalvKooten Textual sequence are continuous variables since we use word embeddings $\endgroup$
    – Franck Dernoncourt
    Commented Sep 21, 2016 at 19:02
  • $\begingroup$ Yea, but I don't see how embeddings can deal with actual continuous variables. For each "value" you would need a different "direction". Even though semantically the number 2 and number 3 might be very close, in reality, the output value should be 2 for 2 and 3 for 3, not 2.5 for both (which would happen if you consider them semantically similar. If you don't use a continuous variable as "input", then you would need an output class for each number.... $\endgroup$
    – PascalVKooten
    Commented Sep 21, 2016 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.