When a user searches for an item to purchase on a retail website they can input some features of their desired item to narrow down their search results. This produces a list of items that match their search. They can subsequently click on and 'view' some of these items that tickle their fancy.
However, I only have data for their search parameters (the features and their values) and also the items they view afterwards. I do not have the list of items from the search. Given this, how would I go about evaluating a machine learning algorithm that tries to learn what items a user views, given this lack of data?
E.g. What would you separate into training/test data? Would the domain space of items be all possible items on the retail website which leads to a huge imbalance in classes?
I have thought about generating synthetic data from the search parameters or sampling across different users, however both methods seem ugly and unsound.