1
$\begingroup$

I am having a hard time thinking and finding prior reasearch about how to include higher level features in neural networks. Here is a couple silly example of what I mean...

Let's say I am building a convolutional neural network to classify if an image has a boat in it or not. How would I include, non pixel level, static features such as "This image was scraped from a site with a .boat TLD".

Let's say I am building some type of recurrant neural network to determine the next word in a sequence of text. How would I include information like "The text I am trying to predict on is coming from the sports section of a newspaper."

I can think of a few ways to do this, for example concat the higher level features represented as word or pixel in the input data but that seems a little crude. Perhaps stacking models that handle observation level features is an answer? Is there a cononical way of handling this?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Insert the data in the fully connected layer(s). This requires a flexible neural network library though, or one you coded your own. $\endgroup$
    – Thomas Wagenaar
    Commented May 12, 2017 at 15:30
  • $\begingroup$ Do you mean concatting the output from the, in the second example, LSTM layer with the features I described as inputs to the fully connected layer. $\endgroup$
    – Bruce Pucci
    Commented May 12, 2017 at 15:37
  • 1
    $\begingroup$ I was thinking more of the first example. You convolute and pool the image of the boat a couple of times, until you have reached the fully connected layer. You concat the output of the last convolution/pooling layer with the additional data into the fully connected layer. For the second example, exactly as you just said :), so yes! $\endgroup$
    – Thomas Wagenaar
    Commented May 12, 2017 at 15:44

1 Answer 1

Reset to default
3
$\begingroup$

So to extend my comment in an answer: you need to incoperate that 'extra' data where it is necessary. As you said, you can't incoperate this in your pixel data, so you have to insert it after the convolutional/pooling layers. Add the extra information in the fully connected layers:

enter image description here

For the LSTM example you provided, you can do as you proposed yourself:

...concatting the output from the, in the second example, LSTM layer with the features I described as inputs to the fully connected layer.

But for the second example you might want to include it before the LSTM layer, if the text your analysing comes from a comic, the network might decide this must have an influence on the gates in the LSTM block for example. It's just a matter of what seems to work best.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.