I have a data source containing millions of documents from a wide variety of business domains. We've aggregated the data such that we can easily find information using natural-language search queries.
I'm looking to run a data science competition internally, and naturally the challenge is going to revolve around text analysis. A recommender engine seems to be the most logical topic for this challenge, given our platform is a search engine.
Unfortunately, it seems the challenge will likely be of an unsupervised-learning nature, as we have no user data yet for our application, and thus have really no way to gauge how effective really the produced recommendations would be.
I see three options:
Leave a random feature (i.e. a random word) out of the dataset and have competitors predict the occurrence of that feature. Basically treat some feature as the dependent variable.
Use a panel of subject-matter experts (say, five) to subjectively judge the accuracy and relevancy of the recommendations, effectively answering the question, "How close are these recommendations to the actual intent of the user?"
Choose a different topic for the challenge; unsupervised learning does not lend itself well to data science competitions.
My question for you: how would you approach quantifying predictions/results in this challenge? Having run many of these challenges previously, in my spare time, I know how much an objective leaderboard matters for competitors. FWIW, I've also read this post, but I think the goal of this project (i.e. identify the most relevant search results) is about as well-defined as it can be given the unlabeled nature of the data, and therefore there ought to be some metric or scoring approach that should do the trick.