4
$\begingroup$

I am working on a project where I have 1024x1024 brain images over time depicting blood flow. A blood flow parameter image is computed using the brain images over time, and is off the same dimension (1024 x 1024). My goal is to train a neural network to learn the mapping between the brain images over time and the blood flow parameter image. Essentially, I want to feed in the time-series of images (brain scans), and have the neural network output a blood flow parameter image.

I've looked into current CNN architectures, but it seems like most research on CNNs is either done for classification on single images (not images over time) or action recognition on video data, which I'm not sure my problem falls under. If anyone can provide me with any insight or papers I can read on how to train a model on temporal data, with the output being an image (rather than a classification score), that would be immensely helpful.

I also recently ran across a blog article discussing CNN_LSTM networks. It seems like this could potentially be a good fit, if anyone has any input on this that would also be fantastic. Thanks in advance.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

If I understand correctly, it seems that you want a neural network that maps a sequence of (brain) images to a single (blood flow parameter) image.

As you mentioned, a CNN LSTM is a good start. Without a reference, this term may have different interpretations. For your task, I would suggest the ConvLSTM which uses convolution inside the LSTM cell. This gets you image sequence to image sequence. You could simply use the output from the last LSTM cell as your prediction. In encoder-decoder situations, the output of last LSTM cell of the encoder is used as embedding of the input sequence.

There are fully convolutional neural networks which learn image to image mappings. You could apply this to the embedding produced by the output of the final LSTM cell, as described above.

Finally, the LSTM component could be swapped out with an attention mechanism for a convolutional attention network. Attention is often replacing RNNs.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks! I've found those resources really helpful. Right now I seem to be headed towards a direction where I can map images to images, but I'm not sure how to map a sequence of images to one image. Would this be where the encoder-decoder framework comes in, to help map inputs to different sized outputs? $\endgroup$
    – Ebrahim Feghhi
    Commented Aug 11, 2019 at 6:55
  • $\begingroup$ @EbrahimFeghhi An encoder (like ConvLSTM) would output a sequence of images. One technique is to use the hidden state and/or the output of the last step of the encoder as a representation of the whole sequence, as shown in Fig 1 of the encoder-decoder link above. You could use this representation as your output or as an intermediate value which you could pass to more conv layers. I recommend the latter. $\endgroup$
    – bjschoenfeld
    Commented Aug 12, 2019 at 17:55
  • $\begingroup$ This may be a good explanation of the many-to-one RNN use case: karpathy.github.io/2015/05/21/rnn-effectiveness $\endgroup$
    – bjschoenfeld
    Commented Aug 21, 2019 at 17:51
  • $\begingroup$ Thank you so much!! You have been immensely helpful. I ended up implementing a network with convlstm finally connected to a conv2d layer, and it seems to be working well. Thanks again :) $\endgroup$
    – Ebrahim Feghhi
    Commented Aug 23, 2019 at 1:41
  • $\begingroup$ I am glad I could help. If this answers your question, please mark it as accepted. :) $\endgroup$
    – bjschoenfeld
    Commented Aug 23, 2019 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.