# Data Conversion to Standard data format in hierarchical Dirichlet process

I'm trying to test the performance of posterior inference on a set of documents with hierarchical Dirichlet process for topic modeling. How can i convert my data (document) to standard data format ?

such as this command :

[M] [term_1]:[count] [term_2]:[count] ...  [term_N]:[count]


where

[M] is the number of unique terms in the document
[count] associated with each term is how many times that term appeared
in the document.


Seems like you need help with some basic parsing of a text. I'm not sure this is the right place to ask but I'll give you a quick answer here as I can't migrate questions between pages.

Here is a simple solution in python:

#!/usr/bin/env python
import re
from collections import Counter

# change to text = input() to read from STDIN instead.
text = """This is a text where there are several words and a few repeating
terms. The text is nonsensical but this is not of interest."""

# this splits the text into lower case tokens and uses a
# Counter to count the occurrences.
terms = Counter(re.split("[\s\.]+", text.lower().strip('.')))
M = len(terms)

# Print the data on the requested format
print(' '.join([str(M)] + ['{0}:{1}'.format(k, v) for k, v in terms.items()]))


This outputs:

19 a:2 of:1 terms:1 is:3 there:1 this:2 text:2 but:1 and:1 the:1 few:1 words:1 nonsensical:1 are:1 repeating:1 several:1 not:1 where:1 interest:1


The script is quite simple and depending on the amount of data you are processing it might not be sufficient.