5
$\begingroup$

So I have a data set of articles like so.

‘Difference of Python and Ruby’    537
‘Advanced Tutorial on Python’     1438
‘HTTP library’                     134

The numbers next to them are click data. I am looking to do regression analysis between the words in each of the titles and the number of resulting view the article received, i.e. to draw conclusion on which words are best to use when titling an article for a specific community or group.

Also, I would love to hear other ideas as well as to what other cool conclusion I could draw from such data.

Edit

Requested Data I also added unique visits data as well.

[('Travis CI: Announcing Python and Perl support on Travis CI',
  {u'clicks': 81, u'unique': 74}),
 ('Software Maniacs blog \xc2\xbb Django as a micro-framework',
  {u'clicks': 333, u'unique': 318}),
 ('Interpreted Languages: Perl, PHP, Python, Ruby (Sheet One) - Hyperpolyglot',
  {u'clicks': 236, u'unique': 221}),
 ('PyCon 2012 has gone mobile on Guidebook!', {u'clicks': 1, u'unique': 1}),
 ('Twitter', {u'clicks': 17, u'unique': 17}),
 ('Static Modification of Python with Python: the AST Module - Blueprint Forge.',
  {u'clicks': 122, u'unique': 109}),
 ('Python for iOS - A Python interpreter and development environment for the iPod Touch, iPhone, and iPad',
  {u'clicks': 151, u'unique': 142}),
 ("Vivek's blog - Roll your own autocomplete solution using Tries.",
  {u'clicks': 211, u'unique': 203}),
 ('Twitter', {u'clicks': 4, u'unique': 4}),
 ('Ian Bicking: a blog :: Python Application Package',
  {u'clicks': 114, u'unique': 111}),
 ('PEP 414 -- Explicit Unicode Literal for Python 3.3',
  {u'clicks': 50, u'unique': 50}),
 ('Static Modification of Python with Python: the AST Module - Blueprint Forge.',
  {u'clicks': 0, u'unique': 0}),
 ('Requests: HTTP for Humans \xe2\x80\x94 Requests 0.10.6 documentation',
  {u'clicks': 18, u'unique': 17}),
 ('Haskell, Python and readability : programming',
  {u'clicks': 246, u'unique': 236}),
 ('PyCon 2012 has gone mobile on Guidebook!', {u'clicks': 40, u'unique': 38}),
 ('Pycoders Weekly (pycoders) on Twitter', {u'clicks': 12, u'unique': 12}),
 ('SWIX social media optimization, and social network marketing ROI',
  {u'clicks': 1, u'unique': 1}),
 ('PEP 414 -- Explicit Unicode Literal for Python 3.3',
  {u'clicks': 0, u'unique': 0}),
 ('[Python-Dev] PEP 414 - Unicode Literals for Python 3',
  {u'clicks': 82, u'unique': 82}),
 ('Google App Engine Blog: Announcing the General Availability of the Python 2.7 Runtime for App Engine',
  {u'clicks': 57, u'unique': 56}),
 ('What\xe2\x80\x99s new in Celery 2.5 \xe2\x80\x94 Celery 3.0.21 documentation',
  {u'clicks': 237, u'unique': 231}),
 ('Python Closures and Decorators (Pt. 2)', {u'clicks': 59, u'unique': 57}),
 ('Python for iOS - A Python interpreter and development environment for the iPod Touch, iPhone, and iPad',
  {u'clicks': 0, u'unique': 0}),
 ('Travis CI: Announcing Python and Perl support on Travis CI',
  {u'clicks': 1, u'unique': 1}),
 ('Quiz & Learn Python - Learn Python on Your Mobile Device',
  {u'clicks': 259, u'unique': 244}),
 ('Python Closures and Decorators (Pt. 1)', {u'clicks': 1, u'unique': 1}),
 ('Python Closures and Decorators (Pt. 1)', {u'clicks': 219, u'unique': 207}),
 ('Trie - Wikipedia, the free encyclopedia', {u'clicks': 18, u'unique': 15}),
 ('Ian Bicking: a blog :: Python Application Package',
  {u'clicks': 0, u'unique': 0}),
 ('Software Maniacs blog', {u'clicks': 42, u'unique': 41}),
 ('Quiz & Learn Python - Learn Python on Your Mobile Device',
  {u'clicks': 1, u'unique': 1}),
 ('Python Closures and Decorators (Pt. 2)', {u'clicks': 0, u'unique': 0}),
 ('SWIX social media optimization, and social network marketing ROI',
  {u'clicks': 71, u'unique': 69}),
 ('Twitter', {u'clicks': 6, u'unique': 6}),
 ('Software Maniacs blog \xc2\xbb Django as a micro-framework',
  {u'clicks': 4, u'unique': 4}),
 ('Haskell, Python and readability : programming',
  {u'clicks': 0, u'unique': 0})]
$\endgroup$
7
  • $\begingroup$ I tried to write a more specific title to attract attention. Please let me know if I misunderstood your question. $\endgroup$ – Gala Aug 7 '13 at 9:15
  • $\begingroup$ @GaëlLaurans totally correct. $\endgroup$ – myusuf3 Aug 7 '13 at 16:53
  • $\begingroup$ Can you make the data available? :P $\endgroup$ – zx8754 Aug 7 '13 at 17:02
  • $\begingroup$ @zx8754 yes I could incoming. I would love how to go about it myself using that sample there as an example $\endgroup$ – myusuf3 Aug 7 '13 at 17:07
  • 1
    $\begingroup$ @MichaelLugo +1 on your irony detection -1 on your helpfulness. $\endgroup$ – myusuf3 Aug 7 '13 at 18:15
6
$\begingroup$

The first thing you need to do is make a matrix, where each column is a word and each row is a document. The entries in this matrix will be counts of that word's frequency in the document. This is known as the bag of words model. You might consider removing words that occur in almost all documents as well as words that occur in almost no documents. It looks like you are using python, and I'm sure python has tools for doing this.

Now, it's a simple matter of doing a regression, where your matrix of word frequencies is the x variable, and number of clicks is the y variable. Note that you're looking at count data, so you probably should be doing a poisson regression. In R you would do this by specifying family='poission' to the glm function.

Additionally, you might want to include an "offset" or exposure variable. This accounts for the fact that different documents might not have shown up the same number of times, and thus might not have had the same number of chances to be clicked on. In R, you would do this by adding an offset() to your regression formula. The offset has a known coefficient of 1.

It also might be a good idea to include a lasso penalty in your regression model. This will remove some words from the equation if they do not contribute a lot to the model's accuracy. In R you would do this with the glmnet package.

Finally, if you have a large number of documents and potential words, it might be a good idea to use a sparse matrix to represent your data (e.g. using the Matrix package in R). glmnet can operate on sparse matrices, which will save you a lot of computation time.

The end result of this process will be a regression model based on each word's presence or absence in the title, where the coefficient represents how many clicks that word adds to the document.

$\endgroup$
7
  • $\begingroup$ awesome! I am using pandas in place of R. Thanks for the pointers on where to begin here. $\endgroup$ – myusuf3 Aug 7 '13 at 19:55
  • $\begingroup$ @myusuf3 Then you should probably check out NLTK and scikit-learn. I think there's a lot of tools there for representing text and using it in regression. $\endgroup$ – Zach Aug 7 '13 at 19:59
  • 2
    $\begingroup$ sklearn has lasso: scikit-learn.org/0.13/modules/generated/… $\endgroup$ – David Marx Aug 7 '13 at 22:02
  • $\begingroup$ @DavidMarx any suggestions on how to do the rest with scikit-learn $\endgroup$ – myusuf3 Aug 8 '13 at 19:30
  • $\begingroup$ As Zach suggested, you'll probably want to use NLTK for the text processing. I believe scipy or numpy has a sparse matrix data type. Poke around in the documentation a bit. $\endgroup$ – David Marx Aug 8 '13 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.