For a game, I am currently trying to train a network that would be able to detect hands in (close to) real time from a webcam feed. What I was hoping to do is train the output network to emit high values for regions that contain hands, I will post-process that with a non-maximum suppression / mode-finding to find the actual location.


I have recorded and annotated some data of myself waving my hands around. This data is the source for generating training targets. I have currently opted for a simple blob around the hand center with a radius corresponding to the hand size. I recorded some data with me sitting in front of the webcam, and some data from further away. I am planning to record much more data, but before investing a lot of time in that, I want to have a better grasp of the requirements, so I started experimenting with a quick and dirty dataset first.

Fig 1: Examples of annotated data:

Fig 1

Fig 2: Example of target image with blobs where the hands are:

Fig 2


To start simple and naively, I took the first two Conv Blocks with weights from VGG16, and added another trainable Conv Block on top of that, finishing with a 1x1 Conv2D layer to collapse all the filter weights into a 2D output image.

conv2d_1 (Conv2D)            (None, 64, 240, 426)
conv2d_2 (Conv2D)            (None, 64, 240, 426)
max_pooling2d_1 (MaxPooling2 (None, 64, 120, 213)
conv2d_3 (Conv2D)            (None, 128, 120, 213)
conv2d_4 (Conv2D)            (None, 128, 120, 213)
max_pooling2d_2 (MaxPooling2 (None, 128, 60, 106)
conv2d_18 (Conv2D)           (None, 512, 60, 106)
conv2d_19 (Conv2D)           (None, 512, 60, 106)
conv2d_20 (Conv2D)           (None, 512, 60, 106)
max_pooling2d_8 (MaxPooling2 (None, 512, 30, 53)
conv2d_21 (Conv2D)           (None, 1, 30, 53)

Fig 3: Using this method on the 'nearby' dataset, the results are actually decent: Fig 3

It looks like the network is really marking the hands, and the output would certainly be good enough to localise hands with a proper mode-finding method. I am also excited that this works with relatively few layers, increasing the possibility of running this in real-time.

Question: Appropriate loss function

For the nearby dataset, a significant portion of the pixels in the image consists of hands, but for the 'faraway' data, the distribution is actually really skewed. See Fig 2, only a few pixels have positive values, the majority is zeroes. What is an appropriate loss function for this distribution of the target data as above? I.e. a black (zero) image with sparse white (positive) areas. Since only a few pixels have positive values, I feel like MSE will encourage the network to generalise and output low values everywhere.

The second part of the question is about "ambiguous" areas. In annotating the data, I have distinguished between "Open and Facing" hands, and "Other hands" (hands that are far away, fists, or other angles, see image below).

Fig 4: "Other Hands" Fig 4

I do not necessarily need the network to recognise these, but I also do not want to 'punish' the network for highlighting them. I suspect that training the network will be easier if the network is allowed to also give (smaller) positive outputs for Other Hands, because they will carry some of the features that also characterise Open Hands (e.g. skin color). In the final network output, I am be satisfied if the Open Hands just light up more brightly. Since I have already distinguished the classes in my dataset, I figured I could somehow tell the optimiser that it doesn't matter what it outputs for these "Other Hands", as long as it gets the Open Hands right.

I also posted this question on the fast.ai forum, I will share a summary of responses there.


First, I suggest using binary crossentropy loss instead of MSE. Binary crossentropy usually does much better at binary classification than MSE. In addition, you can usually interpret the predicted values as probabilities.

In order to ignore other hands, the simplest way would simply to apply a mask to mask out the loss on those locations. So the full loss computation would look something like

loss = reduce_sum(cross_entropy(predicted, labels)*(1.0-other_hands_mask))

  • $\begingroup$ With binary crossentropy, should I modify my training data so that it outputs binary values for each pixel? I currently create a radial gradient around the hand with values of 1 in the center, fading out towards 0 (Fig 2 and Fig 3) $\endgroup$ – noio Aug 22 '17 at 7:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.