Metric for unsupervised recommender-system competition?

Question

I have a data source containing millions of documents from a wide variety of business domains. We've aggregated the data such that we can easily find information using natural-language search queries.

I'm looking to run a data science competition internally, and naturally the challenge is going to revolve around text analysis. A recommender engine seems to be the most logical topic for this challenge, given our platform is a search engine.

Unfortunately, it seems the challenge will likely be of an unsupervised-learning nature, as we have no user data yet for our application, and thus have really no way to gauge how effective really the produced recommendations would be.

I see three options:

1. Leave a random feature (i.e. a random word) out of the dataset and have competitors predict the occurrence of that feature. Basically treat some feature as the dependent variable.

2. Use a panel of subject-matter experts (say, five) to subjectively judge the accuracy and relevancy of the recommendations, effectively answering the question, "How close are these recommendations to the actual intent of the user?"

3. Choose a different topic for the challenge; unsupervised learning does not lend itself well to data science competitions.

My question for you: how would you approach quantifying predictions/results in this challenge? Having run many of these challenges previously, in my spare time, I know how much an objective leaderboard matters for competitors. FWIW, I've also read this post, but I think the goal of this project (i.e. identify the most relevant search results) is about as well-defined as it can be given the unlabeled nature of the data, and therefore there ought to be some metric or scoring approach that should do the trick.

Answer 1

Out of the three options, let's first talk about option 2 and option 3 first, then option 1.

Option 2: Panel expert judgement

No. Never trust subject-matter experts on their judgement about anything quantitative. This applies to even ML experts on their topic-of-interest ML models' performance metrics. In fact, the more the experts know about the subject, the more biased they are about that subject. In addition, experts tend to be over-confident on their ability to avoid biases, which makes it even worse. That's why almost all companies insist on A/B test before fully deploying models.

Listen to how Gary King explains a version of it: https://youtu.be/rBv39pK1iEs?t=594

Option 3: Accept that it's unsupervised and look for other tasks for the competition

Let's assume that we accept the fact that we do not have any label data and we are doing some sort of clustering. Although there are some metrics that define how well documents cluster based on the words in them, it boils down to having an algorithm that optimizes whatever metrics you pick directly. So at the end the competition becomes who's designing a better EM algorithm for the chosen metric, which is pretty lame.

So yes, probably it's best to move on to other datasets.

Option 1: Make it supervised by creating labels

This is the only option assuming we do not want to give up on the dataset.

One option is to leave the title, and/or first paragraph, and/or last paragraph out, and extract some keywords (e.g. simply TF-IDF) from them as document tags, and make it a tag prediction task.

The another option is to think about some external attributes of the documents that you can harvest with a reasonable amount of time and effort. This most likely depends on the kind of documents you are handling. For example, if the documents are financial reports of public companies, you can check the price change for their stocks after the date of the report release. If it's academic papers, you can try to extract their number of citations via Google scholar and have a task to predict citation at 90 day after publication. It heavily depends on context and creativity.