I am trying to make a computer vision system which will be able to detect and track objects of interest. This will require (1) detection functionality to notice the object when it appears (2) classification functionality to know the object is indeed of the correct object class and (3) tracking capability to follow the object. Also importantly, the pipeline will need to process huge images (~25megapixels) in close to real time.

I am trying to find the best data/algorithmic pipeline to accomplish this. I know for example I can just use a deep network for oject detection, which will also implicitly do classification and tracking. However, at 25megapixels that will run extremely slow.

Initial Approach:

I am thinking to use some very fast algo for background subtraction to identify putative patches of the image which are changing. If I make the assumption the camera is stable, I can just use vanilla background subtraction. Otherwise, I am not sure how to do this.

The putative patches will go to a fast deep NN for classification (such as MobileNetV2).

The patches which are identified to be of the class of interest will be sent to a correlation filter based tracker such as SiamFC.

In summary: background subtraction --> classification --> tracking.

Note only background subtraction operates on the whole image. The other 2 only operate on local patches.

Is this a sensible approach? Are there any better approaches?

Also, does this problem have a name? I havent been able to find anything in the literature that specifically deals with this? Any links to articles/discussion of this would be great.



1 Answer 1


I work on similar project, few hints:

-You talk about tracking -> you have a video -> how many FPS (frames per second) does it have? Each frame is 25MB? Thats huge, neural nets does not need such high quality video, is there a possibility to compress the images to speed things up? Most of the common neural nets are trained on relatively small images, like 256 x 256 pixel or so.

-Yeah, you can do background substraction to detect object (check which pixels / areas differ from background by some threshold and cluster them by something like mean shift clustering). It will not have the highest accuracy, but may be good for your case.

-You can use neural nets (or other techinques, e.g. HOG + SVM) for object detection. (YOLO is good and slow, variants of Single Shot Detector or tiny-YOLO are faster, you can find many pretrained model) . Hovewer, the problem of tracking is more complicated than classification, you need to pair the same object between consecutive frames, and / or decide that new object enters the scene. For simple problems, you can match the object by distance of their bounding boxes, maybe using Kalman filter. If you expect the object to move closely each other, the problem become harder. One way is to extract some descriptors of the object (e.g. average color) and match them also by this information.

-you may be interested in this paper: https://arxiv.org/abs/1703.07402 which deals with tracking people on the video with descriptor obtained by another neural net.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.