The question is edited for clarity after tchainzzz's comments about meta-learning.
Let's say we have 10,000 pet pictures and 10,000 kids. Each kid is presented with 10 randomly picked pet pictures at a time. Each time, they have to pick the one picture that they like best. Our goal, during inference, is to predict probabilities on which picture (from 0 to 9) the (same) kids will pick. My struggle is how to construct a NN to make this classification.
Paths I've been thinking about or tried:
I have created embeddings for kids and pictures using a (Netflix competition winner style) factorization method. The embeddings are pretty good: Visualizing the picture embeddings in a projector, similar pets are grouped together.
The first thing I tried was to concatenate the embeddings of the 10 pictures and feed this together with the embedding of the kid. The output layer a softmax and CE loss. But it doesn't work - I guess it's to difficult for the model to "understand" where one picture embedding starts and another stops, and to relate each of the embeddings to the 10 categories in the output layer.
tchainzzz pointed me in the direction of meta-learning, including few-shot learning (before I had clarified my case). But these methods are mainly intended for classifying the entities (is the pet a dog or a cat?) and they are intended for limited training sets. In our case, we're not classifying the pictures (we already know which ones are cats and which ones are dogs) and we have ample training data.
Why not use metric learning with siamese networks? I don't think it will work here, because this method assumes that there is one ideal pet that each kid would select, and we just need to figure out which picture is more like that ideal pet. But we don't have an ideal pet for each kid, only the previously performed selections.
Why not use some kind of ranking solution? (We could probably create a system, like elo chess ranking. Every time a kid selects a picture, that picture would get a higher ranking, particularly for that kid, and more so if the competing picture already have a high ranking.) Because that's not a neural network classification architecture. I can add such a ranking as a feature, but the question is how to create an NN model so that it "understands" that the classification should happen from a menu of 10 available dishes.
There is, however, elements from 'siamese networks' that I've been thinking about. Not the metric part of siamese architechture, but the 'shared weights' part: A possible solution to my problem could be that I insert the embedding for the kid next to the embedding of one picture (i from 0 to 9) into 10 siamese twin networks (i from 0 to 9) sharing the same weights. Each twin would have one output mapped to 10 classes in a softmax layer. (The softmax layer is on the outside of the siamese part of the network.) I have tried this quickly, without much luck. But so far, this is my best idea and i'm continuing to work in this direction.
Any further advice or ideas would be welcome!