I've been studying the bag of features (BoF) methods. I understand the basics and how it's done but I think I still am not good at the details. So here's my understanding of such methods:

Suppose you have a data base of images and you need to classify them. OK, then you must select local features in the first place. I read that there's no general rule on how they're selected, it highly depends on the application. Then, considering each of these local features is called "visual vocabulary", the same steps as bag of words is applied here, too. In bag of words, we count the important words of a document using a histogram.

Here are my questions:

  • When your data base of image is relatively large and you have no information of the images individually, how are those local features selected? How do you know which parts of the image contains information and are thus important to us?
  • I read that these methods are simple because spatial features are ignored. Don't we lose information this way, or even, do we have enough information?

If you could suggest some good reads on the topic, I'd really appreciate it but please don't publish Google results, I've read most of them (if not all of them!).

In case someone is interested to know why is the word "bag" used; that's because they're unordered collections.

  • 1
    $\begingroup$ It seems to me this question is more suitable for Cross Validated. $\endgroup$ – Kaveh Feb 4 '14 at 18:37
  • $\begingroup$ I agree with @Kaveh, if you'd like your question migrated then just flag it as such or comment here and I will take care of it. $\endgroup$ – Artem Kaznatcheev Feb 6 '14 at 3:33
  • $\begingroup$ @ArtemKaznatcheev: I think it's CS specific but still what choice do I have? please move it to CV if you think it'll be answered there. $\endgroup$ – Gigili Feb 6 '14 at 6:58

First of all, you should distinguish between keypoints (feature points) and their descriptors (feature vectors constructed on the base of an image patch taken around corresponding keypoints). In theory, optimal detection and description of keypoints should be deeply connected. However, in practice, one can detect keypoints and construct their descriptors using almost unrelated methods.

Keypoints are the most unique (or informative) points and most stably detectable points on images. Of course, informativeness of points depends on image type under consideration. For ordinary images, informative and stable points lie on borders of uniform regions (since our visual world approximately consists of a set of smooth surfaces). Consequently, most detectors of keypoints are based on maxima of partial derivatives (gradient, Laplace operator, etc.). Of course, this is a simplification, so there are many different implementations of these detectors, which have somewhat different performance (and speed) on different images. If you don’t want (or cannot) to select more suitable detector, the simplest way is to take any general detector, e.g. on the base of difference of Gaussians.

If your images are very specific (very unnatural), the only way is to automatically construct a good detector using the criterion of its informativeness. Different approaches to this problem exist originating from statistics and information theory. Or one can try adopting something like convolution autoencoders with pooling.

The same is true for descriptors, but their influence on performance more significant, and methods for constructing features are even more diverse. However, considerable part of hand-crafted feature transforms is very similar to features that can be obtained automatically by PCA, ICA, autoencoders, etc. Hand-crafted features can be calculated faster (due to code optimization), and don’t require off-line learning. But if your images are of really unknown nature, then it is better to learn feature transforms from some training set of images. However, there is a questing concerning their invariance. Most methods with hand-crafted feature transforms perform affine (or rotation and scale) normalization of image patches before calculating features in order to achieve invariance to spatial transformations. Right now, no methods exist, which can learn arbitrary invariants.

Indeed, in bags of visual words spatial arrangement of keypoints is ignored. This makes comparison of images very fast. And, indeed, we lose valuable information this way, and in some cases remain information could not be enough for good recognitions. However, an alternative solution is to perform keypoint matching, which is much slower. Well, there are also intermediate solutions, in which some spatial information is incorporated, but full point-to-point matching is not needed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.