0
$\begingroup$

I need to detect persons in a scene.

  • I have a 16 bits depth image of that scene (640, 480)
  • I have a 16 bits thermal image (80, 60) of the same scene (slighly different point of view)

I resize these two images to (300, 300) for the SSD input. I label on the thermal images since it is easier to see the silhouette.

(Note that I can stack the 2 channels because I know the rotation/translation matrix between those 2 images)

Can I use classic object detection models like SSD or Faster-R-CNN by stacking those images into one input (2, 300, 300) ?

If yes, how should I standardize each channel ?

Or, Is a multimodal architecture, taking 2 distinct inputs and then stacking features, a better way to solve my pb ? Can the multimodal architecture automatically handle the different points of view of my 2 input images ? (I'm not familiar with this second approach, so that's why I ask)

Thermal image :

thermal image

Depth image :

depth image

$\endgroup$
0
$\begingroup$

Stacking the two modes into one should be perfectly fine.

If yes, how should I standardize each channel ?

With modern training techniques, standardization is not too critical -- anything reasonable should work.

Can the multimodal architecture automatically handle the different points of view of my 2 input images

In theory yes, having a multi-modal stereo input would probably perform better, but only if you have a specially designed architecture for it. This stuff gets pretty involved so unless it's mission critical it's probably not worth the complications. I recommend reading the paper "Stereo-RCNN" for inspiration.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ As the convolutions keep localisations of objects across the network, I dont understand how the network can understand that an object , represented by pixel (x, y) in depth channel (a very tiny object indeed, but it is just for the example), and the same object in thermal channel represented by pixel (x+delta, y+delta) are the same. (I do the annotation only the thermal channel) $\endgroup$ – antoine Mathu Dec 3 '19 at 14:50
  • $\begingroup$ @antoineMathu well, as you said yourself -- if you have the ground truth pose of the cameras, and you have the depth of one of the viewpoints, you can simply warp/project it onto the other viewpoint. $\endgroup$ – shimao Dec 4 '19 at 7:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.