6
$\begingroup$

The problem I am trying to solve is, given two images, determining whether they contain the same object or not. Here is an example:

enter image description here enter image description here enter image description here

The first two images contain the same object, while the third image contains a similar, but different object. My goal is for the first two images to be seen as a match, but the first and third (and second and third) being seen as not matching. I want the matching to work in general with any object. It should be able to tell if any two pictures of any two objects are identical objects (not just similar) even if they are taken at different angles, cameras, and lighting conditions.

I've tried using the SIFT algorithm to find keypoints and descriptors. Then using the cv2.findHomography function with RANSAC to get the inliners. This doesn't really work, and I haven't been able to get it to output good keypoints or matches. They are basically garbage.

I've also tried using a pretrained model (like ResNet50) and removing the last few output layers (using include_top=False for the model in the python keras library). Then I can calculate the cosine similarity between the flattened output tensors of the model for both images. This gives good results for determining if two images contain similar objects, but not for determining if they are the same object.

Recently, I've been trying to do transfer learning from the pretrained imagenet model to a siamese convolutional neural network using the triplet loss function. I followed this tutorial https://keras.io/examples/vision/siamese_network/ and it works well. My issue is that I cannot find a dataset that I could use to train it to identify identical objects. I've tried to curate my own dataset, but it is too time consuming for me to do alone.

Overall, I have tried many approaches. The siamese network should work, but I don't have a dataset to train on. Any help would be greatly appreciated.

Update

https://opendata.stackexchange.com/q/20837/32757

Improve this question
$\endgroup$
15
  • 1
    $\begingroup$ I'm not sure if this question is on-topic here. Did you already read the the Help Center page on what questions are on-topic here? If so, can you explain to my why this is on topic? (since I'm not sure). If not, please read it and consider whether your question is on-topic here. If you're not sure where to ask this question, you can ask on meta.stackexchange.com using the [site-recommendation] tag. $\endgroup$
    – starball
    Commented Jan 8, 2023 at 23:39
  • $\begingroup$ Do you think it is off topic? If so, I'll gladly move it to where it is better suited. $\endgroup$
    – NoahGav
    Commented Jan 8, 2023 at 23:42
  • 2
    $\begingroup$ You might look into algorithms for loop-closure detection. This is a similar problem encountered in Visual-SLAM where you want to detect whether you've returned to the same location as before. You may not need to train a big deep model to solve this problem. $\endgroup$
    – Peter
    Commented Jan 8, 2023 at 23:52
  • 2
    $\begingroup$ @noahgav "This is good enough to determine if objects are similar, but not good enough to determine if they are the same" This is the best you can expect from a solution. Obviously your images are taken under different lighting conditions, with different perspective distortions and different background. This means, that the images of the objects are NOT identical. The best you can hope for is a distance measure on images that returns the actual identical objects among the top ranked similar images. $\endgroup$
    – cdalitz
    Commented Jan 9, 2023 at 9:27
  • 2
    $\begingroup$ I think you're on the right track -- triplet networks were devised to help identify whether two images contain the same object (or person). I appreciate that you don't have a large annotated dataset, and that this has impaired the quality of the results. So you've done the next-best thing and used pre-trained networks, but, again, those results have not been sufficiently precise. I think the inescapable conclusion is that either you'll need a larg(er) dataset that pertains to the task you need to solve, or accept lower-quality results. I don't see a path to high quality without relevant data. $\endgroup$
    – Sycorax ♦
    Commented Jan 9, 2023 at 15:58

1 Answer 1

Reset to default
1
$\begingroup$

I would assume that the core part of this problem is very similar to identification of people. I would approach it in the same way and look to ID and facematch for resources that can be used.

If you have a set of keypoints from a reference image then you can check if a similar/close enough set of keypoints exists in another image. This will of course be very difficult if you have a huge range of object types and perspectives, but it should be easier if you have a narrower data range.

If you want this to be universal then of course it will complicate matters a lot.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.