How can I create a training data set for document classification using LSA? I have created a term-to-document matrix and have class labels also. I don't know whether to add these class labels in a term-document matrix or to create anther matrix. I don't know the exact steps to follow to create training data.
You proceed the same way you would without LSA:
- take some labeled corpus (e.g. 20 newsgroups)
- preprocess (remove stop words, stem, etc)
- convert to the vector space model, weight by tf-idf
- do LSA
- build a classifier
So the only additional step here is the 4th one.
Here's an example for scikit learn adapted from here:
dataset = fetch_20newsgroups(subset='all', shuffle=True, random_state=42) labels = dataset.target true_k = np.unique(labels).shape hasher = HashingVectorizer(n_features=opts.n_features, stop_words='english', non_negative=True, norm=None, binary=False) vectorizer = make_pipeline(hasher, TfidfTransformer()) X = vectorizer.fit_transform(dataset.data) svd = TruncatedSVD(true_k) lsa = make_pipeline(svd, Normalizer(copy=False)) X = lsa.fit_transform(X) clf = MultinomialNB().fit(X, labels)
Here's a list of useful links on how to work with textual data in scikit learn: