I want to implement convolutional neural network for extracting real values of the features from the image.

I understand that metric learning network tries to learn the most important things of the objects and then it outputs a vector of the abstract descriptors which should be almost the same for the same objects, but how can I get vector of the most important real-world values(e.g. width, area or color of the some local feature, so changing width of something on the image should change only responsible for it value or values) instead of this abstract vector, using something similar to the ResNet or another architecture?

I already tried to search something about this type of metrics or features extraction but it's nothing.

Also as I said it's not only metric extraction, net should find(as ResNet do it) and output non-abstract descriptors.

I understand that computer sees only pixels and so on, but in the learning process neural net learns the most important for description of the objects metrics, so these metrics should be some real-world values(or could be simply converted into them).

Of course any help or hints from you will be very useful!


This topic is quite broad, so the answer depends on the kind of data you have (labelled, unlabelled, intersecting labels, partial labels). So, actually I could recommend several approaches. Hope it will help.

Assuming you have some kind of unsupervised (like auto-encoders) or semi-supervised CNN and talking about some kind of "context" (compressed features). Learning important features basically means getting rid of noise. The definition of noise depends on the way you regularize or supervise your network (it might be "oversized" features, "changing too fast", "doesn't help classification" etc.), so the the behavior of your CNN.

In order to get real-valued outputs you have to use CNN for regression instead of classification - so no sigmoid "outputs" (for context) then. Another option is to encode "levels" of width as classes (if you don't need precise results).

So now, you have to extract some meaning from it...

In order to return something understandable, the network has to know something about what you actually mean by "meaningful representation". So I guess you need to provide targets for some of your data:

  • you could provide width and other known metrics as a target (you'll have to scale it in advance)
  • you can label existing output features, learned in unsupervised way if they correlate with features of interest. The important thing here is to specify some prior knowledge with regularization (for instance adversarial examples that randomly change irrelevant metrics).
  • if one output-feature represents several features of interest - you can apply ICA (independent component analysis) in order to separate them.
  • If you care about actual objects (want to manipulate with one particular object from your image) - you can subtract feature-vectors (vector-of-interest "minus" vector-extracted-from-irrelevant-data) in order to get only related features.

If you don't provide any targets - you'll have to find correlations with real-data manually.

If you work with CNN-based classifier and want to know where exactly classified object is on the scene, you'll probably need "Scene Labeling" - it labels every pixel with a class - so you can get a complete "mask" for your object (and simply extract metrics from it). Example of adding labels to pixels: Convolutional Networks in Scene Labelling

Another example is cat metrics converted to images. It's based on the adversarial training, so instead of supervision, you could just modify your data in a way that makes features of interest most important.

  • $\begingroup$ [1/2 part] Many thanks for your answer! I found a few interesting things in your answer, but something not clear for me. Let's try to consider this algorithm on the example. Supposably we have task to describe and recognize faces using this algorithm, it almost like metric learning do it, but we need to find and output the most important for description and recognition real values(e.g. distance between eyes, depth of eyes, coords of nose, color of lips etc.) using CNN(or another machine learning method)... $\endgroup$ – don-prog Feb 28 '17 at 23:14
  • 1
    $\begingroup$ Basically, there is two approaches. [1/2]: you do scene labelling and just calculate metrics with formula. For eyes, algorithm can return you two masks, so you can calculate distance between (centroids of masks). When you get mask for lips - you can just calculate average color. And so on...So the first part of your pipeline is scene labeling, second - is hard-coded feature extractors $\endgroup$ – dk14 Mar 1 '17 at 3:07
  • 1
    $\begingroup$ [2/2] Instead of hard coded feature extractors from scene-labels, you can specify targets on your own (it would make sense especially if you need measurements expressed in centimeters - it definitely won't be any precise). Which means that instead of (or better - in combination with) learning pixel-labels you can specify few more output features: lips color (scaled to be in 0..1, or embedded one hot vector for known colors), distance between eyes, depth of eyes etc. So, in that case it's just same old supervized learning and you have to specify distance, color, manually for many examples. $\endgroup$ – dk14 Mar 1 '17 at 3:13
  • 1
    $\begingroup$ @don-prog talking about pipe-line in [2/2], I think combining it with scene labeling will help (as algorithm will learn distinct zones, which is useful for metrics extraction). You can either have two networks in pipeline: 1) extract masks, 2) pass masked values to metrics learners. Or just add more output neurons to your scene-labelling CNN. P.S. Basically, MLP can approximate any function given enough examples, so the only challenge here is to provide it with enough capacity, labels and examples. $\endgroup$ – dk14 Mar 1 '17 at 3:17
  • 1
    $\begingroup$ @don-prog About (1/4), yes it's correct. The problem about extracting most important features is that most important features for machine are features that explain the input data, so set of automatically learned features depends on the data you provide and for denoising auto-encoders - the way you add noise. So basically if you want such encoder to learn about distance between eyes - you have to provide it with examples of image with changed everything but eye distance and make it encode it back to "normal". Or change eye distance and push auto-encoder to "re-encode" it. $\endgroup$ – dk14 Mar 1 '17 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.