0
$\begingroup$

I have a massive database that contains:

  1. abstracts of journal articles about ethics and moral philosophy,
  2. the date that each abstract was published, and
  3. the number of times that each abstract has been downloaded each month since it was first published.

I want use this data to train an algorithm that will predict the number of times that a new abstract will be downloaded per month based upon the text of the new abstract.

I write code regularly, primarily in Python, but I am new to machine learning and I am having trouble figuring out how to approach this project. From my preliminary research, I believe that this is a problem best addressed by a supervised learning method. There are many supervised learning methods, for example, those included in the scikit-learn package. I don't know how to go about assessing which method would be the best to use for my specific project.

What is the best statistical or machine learning approach to this problem, or, what steps can I take to determine what the best approach is?

This is an example abstract from the database:

For millennia, philosophers have speculated about the origins of ethics. Recent research in evolutionary psychology and the neurosciences has shed light on that question. But this research also has normative significance. A standard way of arguing against a normative ethical theory is to show that in some circumstances the theory leads to judgments that are contrary to our common moral intuitions. If, however, these moral intuitions are the biological residue of our evolutionary history, it is not clear why we should regard them as having any normative force. Research in the neurosciences should therefore lead us to reconsider the role of intuitions in normative ethics.

$\endgroup$
3
  • 2
    $\begingroup$ Possible duplicate of How to choose between learning algorithms or stats.stackexchange.com/questions/120337/… or stats.stackexchange.com/questions/35704/… $\endgroup$
    – Sycorax
    Commented Jan 22, 2017 at 1:25
  • 1
    $\begingroup$ Realistically, this is too broad to be answerable, but I can make a couple of general points. 1) There typically isn't a 'best' ML approach that can be picked from your head. People try different things, & spend a lot of time working & thinking about the problem. People typically start w/ what they feel most comfortable w/. 2) Are you sure there's really much signal in your data? ML algorithms aren't magic; they just try to extract the signal that's there (there has to be a signal for that to work). There probably are some predictive buzzwords, but I doubt you'll get much from this. $\endgroup$ Commented Jan 22, 2017 at 1:35
  • $\begingroup$ You should spend a lot of time learning the basics of statistics and machine learning before embarking on a project of this scope. Machine learning isn't just a programming paradigm or something—it's a field unto itself. $\endgroup$ Commented Jan 22, 2017 at 2:49

1 Answer 1

1
$\begingroup$

I recommend you a simple tf-idf system representation of your documents , first calculate TF(Term Frequency) from each of your element and IDF(inverse document frequency) , then tf * idf for each word will give you a weight , now each document is a vector of word features , you have two approach here , 1) Train a classifier with these features + each time every document downloaded as documents feature and predict new document , you can use SVM , and Naive Bayes classifier 2)with some of similarity functions like cosine calculate similarity of new document with all of your documents then assign the most similar one downloaded time to new document, for example you have 3 documents :

1) How to choose a machine learning algorithm -> 4 times downloaded

2) Maradona is best football player! -> 2 times downloaded

3) Tomorrows weather is snowy!  -> 1 times downloaded


new_document -> best machine learning algorithms 

cosine_similarity(1,new) -> 0.95
cosine_similarity(2,new) -> 0.01
cosine_similarity(3,new) -> 0.20

So because the similarity of first one is outnumber others then probably the new document will be download 4 times.

There are so many tricks to improve your performance and accuracy , so its your times to get your hand dirty!

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.