is there any possibility to iterate over the tokens of a text document within RapidMiner? My first try was to window the document after tokenisation. But this seems very complicated.
I'm doing this to simulate the creation of a language model like Katz Backoff and so on. Maybe there is a better way to produce language models but in the moment i can only think of iterating the tokens and calculate the values and store them into an example set.
Thank you
Andreas
What are you trying to do?
The "Process documents from ..." operators can calculate TF/IDF, Term Frequency, Term Occurences and binary term occurences for you if you select the appropriate option.
The result of "Process documents from ..." is a structure where each token (term) an attribute that has a value for each document. You can use the Loop Attributes operator to calculate something for each attribute.
For example, I used Loop Attributes to sum up the TF/IDF of each attribute (not that it makes sense but it illustrates what you can do).
I added the Loop Attributes after the Process Documents step and set the following options:
Attribute filter type: value_type Value type: real (as the TF/IDF vectors are real numbers)
Loop Attributes creates a macro that can be used to reference the attribute currently in the loop.
Inside Loop Attributes I added a Generate Aggregations operator with the following options:
Attribute name: sumatt_%{loop_attribute} Attribute filter type: single Attribute: %{loop_attribute}
The result is a sum attribute for each attribute (token).
