I would like to submit to an expert group a set of images from two genetically different types of plants to see if they can find a difference. I have 20 images for each of the two types. Images are images of cells that are so specific that only a small group of 10 experts are able to see the potential differences (without knowing which type it is).

To compare two products in sensometry, we can use a triangular test. But here, there is a low number of experts and an important variability in the two products, and I can not show (20*20=) 400 couples of images to my experts. To have a powerful test, I choose to work with a two-out-of-five test, which means that I show experts five images in which 3 are from type1 and 2 are from type2. The have to define two groups of images. I chose this test because of its power. The probability to group the images correctly by chance is only 1/10. Because the number of experts is really small, using this test instead of a triangular test or a simple two images test, is in my point of view more powerful.

The variability of intra-type images is quite high. If there is a difference between the two types, it will be small, within this set of 2*20 images. Hence, if there is a difference, I would like to know which images are the most different. I cannot show more than ~50 sets of 5 images to my experts as they do not have a lot of time to perform the analysis. The problem is that there is about 400000 possibilities if sets of 5 images (3 type1 - 2 type2).

I think that choosing randomly my 50 sets among 400000 for each expert is not really representative. It would be more interesting to choose the sets so that each image has been compared to each other, so that I can define if there are similar or different (which can be infered from the 2-out-of-5 grouping). Randomly, I will not be sure that I tested all images against all (at least indirectly) with my only 10 experts. Thus, I decided to create a sample of 50 sets, different for each expert, that maximises the number of couples compared either directly or indirectly. However, with this sampling, some couples of images are more compared than others, which is why I forced my selection so that each couple is compared at least 3 times directly or indirectly.

This seems to be a quite complicated sampling procedure. For me, this is the only one that makes me sure I will be able to correctly compare all my images, in a few shot for each expert, knowing that once I have shown it to my experts, I will not have any other chance to find another expert group. And they won't be able to take more time to do it again.

To summarize, I want to know :

  1. Are my two types different ?
  2. Which cell images are more different than the others ?

Do you think I am going in a too complicated way ? Is random a better solution even if I will be able to only test 50 sets * 10 experts among 400000 possible sets ? How can I be sure not to bias my sets selection procedure ?

By the way, I work with R, but I don't think this is really important for this question

  • $\begingroup$ Making random samples of 50 images per expert makes me wonder what would happen if one expert was given easier or more difficult images to group? So why not pick one sample of 50 images (with a 3:2 ratio of type 1:type2 images) for all ten experts to put into the two groups? $\endgroup$ – IWS Apr 7 '17 at 13:27
  • $\begingroup$ If I do that only once, I would only be able to say that the experts were good or wrong. Then, I would not be able to determine what makes this groups so common or different. With multiple sets, I can calculate sort of a distance between all couples of images and produce a classification based on the matrix of distances. $\endgroup$ – Sébastien Rochette Apr 11 '17 at 16:42
  • $\begingroup$ Do you really think that experts would be overwhelmed by more than 5 images at a time? When sorting specimens, it is often much easier to spot the relevant differences when more material is available. With just 2 specimens in one group and 3 in another, it is really hard to pick out what the relevant features are! I would suggest giving each expert more like 20 images to classify into two sets would be much better (that's still just 20 images per expert, which is fewer than the 5 sets of 5 images that you propose!). $\endgroup$ – Jacob Socolar Apr 11 '17 at 21:11
  • $\begingroup$ Honestly, if an expert is willing to commit spare time to this project, I would imagine that she would be happy to examine and sort all 40 images. Surely that wouldn't take too much longer than sorting groups of 5 images five separate times. $\endgroup$ – Jacob Socolar Apr 11 '17 at 21:12
  • $\begingroup$ My real problem is that I do not want them to be able to well classify the two groups... Indeed, experts say that there is a difference between the two groups but they cannot put a word on what is the difference. (Thus I cannot calculate image analysis indices for greater number of images as I dont know on what to focus). If I give 40 images at the same time, they will probably be able to separate the groups, but they wont tell me what was the difference. If I give 40 sets of 5, I will know which are more different than the others. I then will be more able to describe this difference (I hope). $\endgroup$ – Sébastien Rochette Apr 12 '17 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.