I have a problem that I am trying to solve using data mining techniques.

What is known:

There is 253 1 page documents that belong to 4 exclusive topics "clustering" "classification" "frequent pattern mining" "data cubes". The documents are about application scenarios that these techniques can be used in.

Each document is written by 2 people. Each person writes 4 documents (one for each topic) with a different person. Each person is identified by a unique random 3 digit number. Each documentID is a unique random 4 digit number. Each word within the documents is represented by a unique random 5 digit number. The actual words are never shown.

Available data:

(Collab.txt file is provided with the following in each row: person1, person2, documentID).

(Doc.txt contains in each row: documentID, numberOfUniqueWords, word1, word2, ... , wordn). Note that these are unique words only, repetitions are deleted.

(Vocab.txt contains a list of unique words, the words' numbers actually).

Problem: some random words have been taken out of each document (words are chosen separately in each document). For each document recommend the top 5 words (in descending order of confidence) that you think is missing. So the resulting file would have on each row: DocumentID, word1, word2, ... , word5

(Precision: most weight put on the first word, second word so on)

As well, could the Collab.txt help with the above problem? Is yes, how? If no, why? (This part may be explained in a page... I don't think it would...)

I had listed out all words on the Vocab.txt not in each document as a start but since there are too many words (less than 300 unique words per document while there are a total of 6k words), I tried clustering documents, in case different people use specific words and then find the frequent patterns missing. But this doesn't give much precision.

A random 20% of the ground truth can be used to test the solution, but we never see the actual ground truth, so testing an infinite amount of times would only really test a bit more than 60% of the solution.

I'd like to put this out to the community to see if there may be other more effective solutions to this problem :) put up your ideas and let's see how precisely we can solve this problem!

Thank you for your time and brains

  • $\begingroup$ Any ideas welcome :) $\endgroup$ – Ponnnnn Nov 23 '15 at 19:23
  • $\begingroup$ This question cannot be answered, even if you post it a fourth time... $\endgroup$ – Anony-Mousse Nov 26 '15 at 22:05

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.