1
$\begingroup$

I'm working on a text classification project, and I want to reduce the tf-idf matrix dimension with Principal Component Analysis (PCA) and then train my model with this, which is pretty straightforward. But once I do that with my training set; how do I transform my test set to the same space to where the training set was mapped? This would be simple if I were working with data where the features are fixed (by just multiplying the component matrix times the test data matrix), but in text analysis, the features change every time a document is added.

So my question is: how to do this with text classification? Or should I try another approach?

Improve this question
$\endgroup$
4
  • $\begingroup$ Aren't you transforming all your documents in your training set before training? $\endgroup$ – gung - Reinstate Monica Oct 27 '14 at 1:53
  • 2
    $\begingroup$ Could you elaborate on "the features change every time a document is added"? Are you retraining your model every time you add a new document? $\endgroup$ – Saul Berardo Oct 27 '14 at 13:19
  • 1
    $\begingroup$ What I mean is that, since the TDM has the words on the columns and the documents on the rows, every new document added to a set, whether it's training or testing, will probably change the whole structure of the TDM, because it will most likely have new words that the former TDM didn't have. That's why the features change every time a document is added. I guess I should have said: 'the number of features increases every time a new document is added'. And I'm not retraining the whole model. $\endgroup$ – Mario Becerra Oct 28 '14 at 22:37
  • 1
    $\begingroup$ I guess what I have to do is to ignore the words that weren't seen on the training set and that are on the test set, so this way I can transform the space where the PCA transformed the training set. Any idea on how to do this in Python? $\endgroup$ – Mario Becerra Oct 28 '14 at 22:45
1
$\begingroup$

What I did was discard all the words that are in the test set and that weren't on the train set, and rearrange everything so the order are the same in each matrix.

It can be seen in the following Python code. x_train is a pandas dataframe which contains the training text, and x_test is a pandas dataframe which contains the test text

#Train
tdm = txtm.TermDocumentMatrix()  
for doc in x_train:
    tdm.add_doc(doc) 
# Push the TDM data to a list of lists, then make that an ndarray, which then becomes a DataFrame.
tdm_rows = []
for row in tdm.rows(cutoff = 3): # The setting cutoff=1 means that words which appear in 1 or more documents will be included in the output
    tdm_rows.append(row)        
tdm_array = np.array(tdm_rows[1:])
tdm_terms = tdm_rows[0]
TDM_df_train = pd.DataFrame(tdm_array, columns = tdm_terms)
TDM_df_train = TDM_df_train.reindex_axis(sorted(TDM_df_train.columns), axis=1) #Ordena las columnas en orden alfabético
#Test
tdm = txtm.TermDocumentMatrix()  
for doc in x_test:
    tdm.add_doc(doc) 
# Push the TDM data to a list of lists, then make that an ndarray, which then becomes a DataFrame.
tdm_rows = []
for row in tdm.rows(cutoff = 3): # The setting cutoff=1 means that words which appear in 1 or more documents will be included in the output
    tdm_rows.append(row)        
tdm_array = np.array(tdm_rows[1:])
tdm_terms = tdm_rows[0]
TDM_df_test = pd.DataFrame(tdm_array, columns = tdm_terms)
#Remove from TDM_df_test words that aren't on TDM_df_train
for col in TDM_df_test:
   if col not in TDM_df_train.columns:
        del TDM_df_test[col]
TDM_df_test = TDM_df_test.reindex_axis(sorted(TDM_df_train.columns), axis=1, fill_value=0)

tfidf = TfidfTransformer()
tfidfRedTrain = tfidf.fit_transform(TDM_df_train.values)
tfidfRedTest = tfidf.fit_transform(TDM_df_test.values)

The last lines just turn the TDMs to TFIDF matrices.

This works perfectly fine with small data sets, but now I have a new trouble. Can this be done more efficiently? Because with another dataset I have with 2000 documents, this doesn't work, it just keeps training and never ends. Does anybody know a way to do this in Scala, Spark, Hadoop or something that is faster? Or can recommend me where to do it?

Improve this answer
$\endgroup$
0
$\begingroup$

You have two options.

The first and easiest one is to simply perform the dimension reduction on a matrix composed by training set and test set together. At the end of the process just split the final set between training and test set either by selecting lines or by a random process.

The second option is to multiply your adjusted test data (values of dimensions subtracted by their mean) by the row feature vector you calculated on the PCA of the training set.

If you are not implementing PCA from the ground up, I suggest you choose the first option. However, if you are using a library that provides you the PCA feature vector, then choose whatever works best for you.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I am not sure I understand the second option. The OP's problem is that test data has different features (and different number of features) than the training data. How can the test data be projected on the principal axes of the training set? Or do you suggest something else, and I misunderstood? $\endgroup$ – amoeba Nov 3 '14 at 16:54
  • 1
    $\begingroup$ Unfortunately, you can not. Your model only works for features seen in training step. There's nothing it can say to you about new words because it's never seen them before. New words, mapped into new features, are ignored. If you want your model to consider new words, you have to re-calculate the model including the new sentences. $\endgroup$ – pedro_assis Nov 3 '14 at 17:50
  • $\begingroup$ I am not the OP :) I understand what you saying in the comment, but I still do not understand the "second option" in your answer (which I would otherwise happily upvote). $\endgroup$ – amoeba Nov 3 '14 at 18:06
  • 1
    $\begingroup$ The first option definitely isn't a choice, because that way the train set has information about the test set, so they aren't independent sets. My doubt was how to do the second option, because dimensionalities are different, therefore matrix multiplication can't be done. What I did was discard all the words that are in the test set and that weren't on the train set, and rearrange everything so the order are the same in each matrix. $\endgroup$ – Mario Becerra Nov 6 '14 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.