0
$\begingroup$

I need to detect persons in a scene.

  • I have a 16 bits depth image of that scene (640, 480)
  • I have a 16 bits thermal image (80, 60) of the same scene (slighly different point of view)

I resize these two images to (300, 300) for the SSD input. I label on the thermal images since it is easier to see the silhouette.

(Note that I can stack the 2 channels because I know the rotation/translation matrix between those 2 images)

Can I use classic object detection models like SSD or Faster-R-CNN by stacking those images into one input (2, 300, 300) ?

If yes, how should I standardize each channel ?

Or, Is a multimodal architecture, taking 2 distinct inputs and then stacking features, a better way to solve my pb ? Can the multimodal architecture automatically handle the different points of view of my 2 input images ? (I'm not familiar with this second approach, so that's why I ask)

Thermal image :

thermal image

Depth image :

depth image

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Stacking the two modes into one should be perfectly fine.

If yes, how should I standardize each channel ?

With modern training techniques, standardization is not too critical -- anything reasonable should work.

Can the multimodal architecture automatically handle the different points of view of my 2 input images

In theory yes, having a multi-modal stereo input would probably perform better, but only if you have a specially designed architecture for it. This stuff gets pretty involved so unless it's mission critical it's probably not worth the complications. I recommend reading the paper "Stereo-RCNN" for inspiration.

Improve this answer
$\endgroup$
2
  • $\begingroup$ As the convolutions keep localisations of objects across the network, I dont understand how the network can understand that an object , represented by pixel (x, y) in depth channel (a very tiny object indeed, but it is just for the example), and the same object in thermal channel represented by pixel (x+delta, y+delta) are the same. (I do the annotation only the thermal channel) $\endgroup$
    – antoine Mathu
    Dec 3, 2019 at 14:50
  • $\begingroup$ @antoineMathu well, as you said yourself -- if you have the ground truth pose of the cameras, and you have the depth of one of the viewpoints, you can simply warp/project it onto the other viewpoint. $\endgroup$
    – shimao
    Dec 4, 2019 at 7:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.