3
$\begingroup$

I'm trying to test the performance of posterior inference on a set of documents with hierarchical Dirichlet process for topic modeling. How can i convert my data (document) to standard data format ?

such as this command :

[M] [term_1]:[count] [term_2]:[count] ...  [term_N]:[count]

where

[M] is the number of unique terms in the document
[count] associated with each term is how many times that term appeared 
in the document. 
$\endgroup$
1
$\begingroup$

Seems like you need help with some basic parsing of a text. I'm not sure this is the right place to ask but I'll give you a quick answer here as I can't migrate questions between pages.

Here is a simple solution in python:

#!/usr/bin/env python
import re
from collections import Counter

# change to text = input() to read from STDIN instead.           
text = """This is a text where there are several words and a few repeating
          terms. The text is nonsensical but this is not of interest."""


# this splits the text into lower case tokens and uses a 
# Counter to count the occurrences.
terms = Counter(re.split("[\s\.]+", text.lower().strip('.')))
M = len(terms)

# Print the data on the requested format
print(' '.join([str(M)] + ['{0}:{1}'.format(k, v) for k, v in terms.items()]))

This outputs:

19 a:2 of:1 terms:1 is:3 there:1 this:2 text:2 but:1 and:1 the:1 few:1 words:1 nonsensical:1 are:1 repeating:1 several:1 not:1 where:1 interest:1

The script is quite simple and depending on the amount of data you are processing it might not be sufficient.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.