4
$\begingroup$

Suppose I'm trying to perform supervised classification on a list of birds into either nocturnal or diurnal. The only features I have regarding each bird are

  • its wingspan (float),
  • its colour (multi-class: white, brown, red, yellow, blue, green, black),
  • its diet (binary: carnivore or not),
  • its habitat (multi-class: ocean, river, mountain, desert), and
  • a sound recording of their call/song (time series of floats)

Let's say I want to use a neural network. A feed-forward neural network will work just fine for the first four features. (The wingspan will simply be one of the nodes in the input layer whereas colour, diet, and habitat will be decomposed into several input nodes, presumably via one-hot encoding.)

However, how shall one deal with the last feature, namely the recording of the bird sounds? If this were the only feature, the natural choice would be a recurrent neural network (RNN), but then what should one do with the first four constant features? Clearly, copying them at each recursion of the RNN is both redundant and contrary to the "spirit" of RNNs since, for example, the colour of the bird isn't expected to change from one sound bit to the next.

In other words, how does the architecture of the neural net account for such mixed features? (The same question applies to other classifiers such as support vector machines.)

Improve this question
$\endgroup$
3
$\begingroup$

If you want to use RNNs, just modify the architecture a bit: plug the time series into a few layers of LSTMs or whatever you'd like to use, and then concatenate the resulting hidden states and the discrete features you have into a softmax layer that produces predictions. So, your computational graph would look like this: enter image description here

Alternatively, there is no rule saying you can't include those constant features as input to the RNN.

Improve this answer
$\endgroup$
10
  • $\begingroup$ I'm sorry, I'm not sure what your diagram from the Keras documentation illustrates. Let me get this straight: You're saying I should first train the RNN on the time series using a sequence to sequence model and then feed the output sequence along with the first four features into a feed-forward neural network (FFNN)? So essentially, I'd train twice in a supervised fashion, first on the time series using an RNN and then on the output of the RNN along with the first four features in a FFNN? $\endgroup$
    – Tfovid
    Nov 2 '17 at 14:12
  • $\begingroup$ No, you train once. Your output is the bird class, right? so the gradient flows back to the merge layer, and there it flows into the two input columns. So, for this schema, you'd split your training data in two: the sounds and the discrete features, and instead of calling fit(X_train, y_train) you'd say something like fit([X_train_discrete, X_train_songs], y_train) $\endgroup$
    – user2089357
    Nov 2 '17 at 14:42
  • $\begingroup$ Oh yeah, and you probably don't want an embedding layer there. $\endgroup$
    – user2089357
    Nov 2 '17 at 14:43
  • $\begingroup$ Alright, so you're saying the RNN will take as inputs the songs and then output a fixed-length vector which will then be concatenated with the first four features as the input to the feedforward neural network? $\endgroup$
    – Tfovid
    Nov 2 '17 at 14:47
  • 1
    $\begingroup$ Right---But then it'll have to be the final state of the hidden layer. I see. $\endgroup$
    – Tfovid
    Nov 2 '17 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.