I'm currently trying to train a custom model with tensorflow to detect 17 landmarks/keypoints on each of 2 hands shown in an image (fingertips, first knuckles, bottom knuckles, wrist, and palm), for 34 points (and therefore 68 total values to predict for x & y). However, I cannot get the model to converge, with the output instead being an array of points that are pretty much the same for every prediction.

I started off with a dataset that has images like this: enter image description here

each annotated to have the red dots correlate to each keypoint. To expand the dataset to try to get a more robust model, I took photos of the hands with various backgrounds, angles, positions, poses, lighting conditions, reflectivity, etc, as exemplified by these further images: enter image description hereenter image description here enter image description here enter image description here enter image description here enter image description here

I have about 3000 images created now, with the landmarks stored inside a csv as such:

enter image description here

I have a train-test split of .67 train .33 test, with the images randomly selected to each. I load the images with all 3 color channels, and scale the both the color values & keypoint coordinates between 0 & 1.

I've tried a couple different approaches, each involving a CNN. The first keeps the images as they are, and uses a neural network model built as such:

model = Sequential()

model.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'same', activation = 'relu', input_shape = (225,400,3)))
model.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'same', activation = 'relu'))
model.add(MaxPooling2D(pool_size = (2,2), strides = 2))

filters_convs = [(128, 2), (256, 3), (512, 3), (512,3)]
for n_filters, n_convs in filters_convs:
  for _ in np.arange(n_convs):
    model.add(Conv2D(filters = n_filters, kernel_size = (3,3), padding = 'same', activation = 'relu'))
  model.add(MaxPooling2D(pool_size = (2,2), strides = 2))

model.add(Dense(128, activation="relu"))
model.add(Dense(96, activation="relu"))
model.add(Dense(72, activation="relu"))
model.add(Dense(68, activation="sigmoid"))

opt = Adam(learning_rate=.0001)
model.compile(loss="mse", optimizer=opt, metrics=['mae'])

I've modified the various hyperparameters, yet nothing seems to make any noticeable difference.

The other thing I've tried is resizing the images to fit within a 224x224x3 array to use with a VGG-16 network, as such:

vgg = VGG16(weights="imagenet", include_top=False,
    input_tensor=Input(shape=(224, 224, 3)))
vgg.trainable = False

flatten = vgg.output
flatten = Flatten()(flatten)

points = Dense(256, activation="relu")(flatten)
points = Dense(128, activation="relu")(points)
points = Dense(96, activation="relu")(points)
points = Dense(68, activation="sigmoid")(points)

model = Model(inputs=vgg.input, outputs=points)

opt = Adam(learning_rate=.0001)
model.compile(loss="mse", optimizer=opt, metrics=['mae'])

This model has similar results to the first. No matter what I seem to do, I seem to get the same results, in that my mse loss minimizes around .009, with an mae around .07, no matter how many epochs I run: enter image description here

Furthermore, when I run predictions based off the model it seems that the predicted output is basically the same for every image, with only slight variation between each. It seems the model predicts an array of coordinates that looks somewhat like what a splayed hand might, in the general areas hands might be most likely to be found. A catch-all solution to minimize deviation as opposed to a custom solution for each image. These images illustrate this, with the green being predicted points, and the red being the actual points for the left hand: enter image description here enter image description here enter image description here enter image description here

So, I was wondering what might be causing this, be it the model, the data, or both, because nothing I've tried with either modifying the model or augmenting the data seems to have done any good. I've even tried reducing the complexity to predict for one hand only, to predict a bounding box for each hand, and to predict a single keypoint, but no matter what I try, the results are pretty inaccurate.

Thus, any suggestions for what I could do to help the model converge to create more accurate & custom predictions for each image of hands it sees would be very greatly appreciated.




1 Answer 1


It would be useful to provide the loss over time for both training and testing data set.

From your description, it seems like you can minimize the training loss, but the testing performance is not going well.

If that is the case, try to regularize the model more. One useful approach is doing data argumentation more.

From the comments, it seems we have problems even with training data, then it is not overfitting but under-fitting issue.

Try to increase the network complexity. Also you can read this post

What should I do when my neural network doesn't learn?

  • $\begingroup$ Thanks for the input Haitao! Added an image of that. It gets to those loss values fairly quickly, but doesn't go down any further once it reaches them, and is fairly even between the two. Also, I've considered doing data augmentation, as that's helped with other models in the past, but don't know how that can be done without messing up the correlation between the images and the coordinates. $\endgroup$ Oct 18, 2021 at 13:20
  • $\begingroup$ @SamSkinner can you overfit your training data? If you input original training data, does the output correct? $\endgroup$
    – Haitao Du
    Oct 19, 2021 at 9:31
  • $\begingroup$ Yes, I have the same issue even with the training data, as the loss for both train & test is about the same. It seems the model goes to a catch-all solution that is the average of the coordinate values rather than a custom one for each image. $\endgroup$ Oct 19, 2021 at 10:16
  • $\begingroup$ @SamSkinner see my edits in my answer $\endgroup$
    – Haitao Du
    Oct 19, 2021 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.