Beginner - How can I use ranked values in my Logistic Regression?

Question

I am running a Logistic Regression on some data to predict if a webpage is "good" or "bad". I got the dataset from a finished Kaggle competiton here (train.tsv). I extract the second column of this dataset, the boilerplate text, run TF-IDF, then Logistic Regression as follows :

  import numpy as np
  import matplotlib.pyplot as plt
  from sklearn import metrics,preprocessing,cross_validation
  from sklearn.feature_extraction.text import TfidfVectorizer
  import sklearn.linear_model as lm
  import pandas as p
  loadData = lambda f: np.genfromtxt(open(f,'r'), delimiter=' ')

  print "loading data.."
  traindata = list(np.array(p.read_table('train.tsv'))[:,2])
  testdata = list(np.array(p.read_table('test.tsv'))[:,2])
  y = np.array(p.read_table('train.tsv'))[:,-1]

  tfv = TfidfVectorizer(min_df=3,  max_features=None, strip_accents='unicode',  
        analyzer='word',token_pattern=r'\w{1,}',ngram_range=(1, 2), use_idf=1,smooth_idf=1,sublinear_tf=1)

  rd = lm.LogisticRegression(penalty='l2', dual=True, tol=0.0001, 
                             C=1, fit_intercept=True, intercept_scaling=1.0, 
                             class_weight=None, random_state=None)

  X_all = traindata + testdata
  lentrain = len(traindata)

  print "fitting pipeline"
  tfv.fit(X_all)
  print "transforming data"
  X_all = tfv.transform(X_all)

  X = X_all[:lentrain]
  X_test = X_all[lentrain:]

  print "20 Fold CV Score: ", np.mean(cross_validation.cross_val_score(rd, X, y, cv=20, scoring='roc_auc'))

  print "training on full data"
  rd.fit(X,y)
  pred = rd.predict_proba(X_test)[:,1]
  testfile = p.read_csv('test.tsv', sep="\t", na_values=['?'], index_col=1)
  pred_df = p.DataFrame(pred, index=testfile.index, columns=['label'])
  pred_df.to_csv('benchmark.csv')
  print "submission file created.."

Now, I have three simple beginner questions I was hoping to get some opinions on please.

Firstly, as you can see, I am only using the second column in my Logistic Regression, combined with TF-IDF. How can I edit my code to also use all integer columns?

Secondly, I understand TF-IDF, and I understand Logistic Regression; but how in this code are they combined? I don't understand how the output of a TF-IDF can work with a Logistic Regression - isn't it just a list of the most common words?

Finally , each row in the training data corresponds to some information about a webpage. I have wrote a program to get the Google PageRank of a URL and spit this into a CSV for me. As a Google PageRank is a rank figure how can I use it in my Logistic Regression? Do I need to visualise it on a graph and then select a average value for a "bad" page, an average value for a "good" page and see which is closer, or is there a smarter way to do this?

Thanks very much for all your help. Please ask if you have any questions.

Answer 1

here are the answers

1. TF-IDF returns a numpy array, to append other numerical columns horizontally to the TF-IDF array you can use 'numpy.hstack'

2. For your second question, I will suggest to go through the book 'Introduction to Information Retrieval' by Christopher D Manning and others

3. Your main goal in the competition is to minimize/maximize the evaluation criteria, nothing to do how Google is rating them, as Google rating may depends on some other objectives like click rate, ADD revenue etc...