I'm looking to train a recurrent neural network model with several different types of input vector. The data includes the input vector of an image (307200x1), an input vector of the position/orientation of one object (7x1), and another input vector of the position/orientation of another object. These vectors have nothing to do with each other, so encoding is not really applicable here. I am trying to figure out if I just concatenate these vectors and feed that as an input for the RCNN, or if I should combine 3 RCNNs, one for each vector, somehow. Each of these three vectors are capable of accurately predicting the correct output of the model, but I would like to combine this information to get even more accurate. Thank you for your time.


I assume by RCNN you're talking about a recurrent convolutional neural network. I don't have direct experience with this type on neural network so take my advice with a grain of salt...

The 'convolutional' part of a convolutional NN is to help process the visual data. In a regular NN, there's no reason the system would suspect each entry in the 307200 x 1 vector to be related to each other, let alone be adjacent (or nearby) pixels in a 640x480 grid.

Since the locational data of the two objects has nothing to do with machine seeing and only a little bit to do with which pixels are highlighted, it makes no sense to run them through convolutional layers, so I suspect neither of your proposed solutions would be what you're after.

As to how to construct your neural network, I don't have the experience to say how it would best be done. I'm imagining a NN with a large block of convoluted nodes where the location/orientation data comes in unaltered as a second set of neurons after the convolutional layer but before the layers used for deep learning. Having said that, I have no idea how you'd do that using any of the neural network tools I've worked with.

I know this doesn't fully answer the question, but I hope it helps.

  • $\begingroup$ Thank you! It actually really helped. I just have to find a way to map the input of the object positions to the output of the convolutional layers, into the layer that assumes full connectivity of the neurons. I'm wondering if I would use an RCNN to output the fully connected neurons (with input to RCNN being image) and feed that output into a RNN, adding the position of the object vector on top of the output of the RCNN. $\endgroup$ – user212022 Jun 19 '18 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.