Loop over Tokens in RapidMiner's Text Processing Plugin is there any possibility to iterate over the tokens of a text document within RapidMiner? My first try was to window the document after tokenisation. But this seems very complicated.
I'm doing this to simulate the creation of a language model like Katz Backoff and so on. Maybe there is a better way to produce language models but in the moment i can only think of iterating the tokens and calculate the values and store them into an example set.
Thank you
Andreas
 A: What are you trying to do?
The "Process documents from ..." operators can calculate TF/IDF, Term Frequency, Term Occurences and binary term occurences for you if you select the appropriate option.
The result of "Process documents from ..." is a structure where each token (term) an attribute that has a value for each document. You can use the Loop Attributes operator to calculate something for each attribute. 
For example, I used Loop Attributes to sum up the TF/IDF of each attribute (not that it makes sense but it illustrates what you can do).
I added the Loop Attributes after the Process Documents step and set the following options:
Attribute filter type: value_type
Value type: real (as the TF/IDF vectors are real numbers)
Loop Attributes creates a macro that can be used to reference the attribute currently in the loop.
Inside Loop Attributes I added a Generate Aggregations operator with the following options:
Attribute name: sumatt_%{loop_attribute}
Attribute filter type: single
Attribute: %{loop_attribute}
The result is a sum attribute for each attribute (token).
