I am running a Logistic Regression on some data to predict if a webpage is "good" or "bad". I got the dataset from a finished Kaggle competiton here (train.tsv). I extract the second column of this dataset, the boilerplate text, run TF-IDF, then Logistic Regression as follows :

  import numpy as np
  import matplotlib.pyplot as plt
  from sklearn import metrics,preprocessing,cross_validation
  from sklearn.feature_extraction.text import TfidfVectorizer
  import sklearn.linear_model as lm
  import pandas as p
  loadData = lambda f: np.genfromtxt(open(f,'r'), delimiter=' ')

  print "loading data.."
  traindata = list(np.array(p.read_table('train.tsv'))[:,2])
  testdata = list(np.array(p.read_table('test.tsv'))[:,2])
  y = np.array(p.read_table('train.tsv'))[:,-1]

  tfv = TfidfVectorizer(min_df=3,  max_features=None, strip_accents='unicode',  
        analyzer='word',token_pattern=r'\w{1,}',ngram_range=(1, 2), use_idf=1,smooth_idf=1,sublinear_tf=1)

  rd = lm.LogisticRegression(penalty='l2', dual=True, tol=0.0001, 
                             C=1, fit_intercept=True, intercept_scaling=1.0, 
                             class_weight=None, random_state=None)

  X_all = traindata + testdata
  lentrain = len(traindata)

  print "fitting pipeline"
  print "transforming data"
  X_all = tfv.transform(X_all)

  X = X_all[:lentrain]
  X_test = X_all[lentrain:]

  print "20 Fold CV Score: ", np.mean(cross_validation.cross_val_score(rd, X, y, cv=20, scoring='roc_auc'))

  print "training on full data"
  pred = rd.predict_proba(X_test)[:,1]
  testfile = p.read_csv('test.tsv', sep="\t", na_values=['?'], index_col=1)
  pred_df = p.DataFrame(pred, index=testfile.index, columns=['label'])
  print "submission file created.."

Now, I have three simple beginner questions I was hoping to get some opinions on please.

Firstly, as you can see, I am only using the second column in my Logistic Regression, combined with TF-IDF. How can I edit my code to also use all integer columns?

Secondly, I understand TF-IDF, and I understand Logistic Regression; but how in this code are they combined? I don't understand how the output of a TF-IDF can work with a Logistic Regression - isn't it just a list of the most common words?

Finally , each row in the training data corresponds to some information about a webpage. I have wrote a program to get the Google PageRank of a URL and spit this into a CSV for me. As a Google PageRank is a rank figure how can I use it in my Logistic Regression? Do I need to visualise it on a graph and then select a average value for a "bad" page, an average value for a "good" page and see which is closer, or is there a smarter way to do this?

Thanks very much for all your help. Please ask if you have any questions.

  • $\begingroup$ I've not run logistic regression in Python so I am less familiar than R, but why can't you just set your X variable for rd.fit to multiple columns? It might help if you share what the output of the tfv.fit looks like. I would assume you can create categorical variables for the logistic regression from this output, or count variables. Same for the pagerank variable, I don't see why you can't just include this as an explanatory variable in the regression, possibly transformed. It might benefit you to read more about how logistic regression works both generally, and specifically in Python. $\endgroup$ – Eric Brady Feb 27 '14 at 18:45

here are the answers

  1. TF-IDF returns a numpy array, to append other numerical columns horizontally to the TF-IDF array you can use 'numpy.hstack'

  2. For your second question, I will suggest to go through the book 'Introduction to Information Retrieval' by Christopher D Manning and others

  3. Your main goal in the competition is to minimize/maximize the evaluation criteria, nothing to do how Google is rating them, as Google rating may depends on some other objectives like click rate, ADD revenue etc...

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.