I've noticed that many pattern predictors initially use the solution of "always pick the most common thing" and sometimes struggle to ever get out of local holes like that for a while.

A simple example is text: at the character level "e" is very very common while at the word level words like "the" and "a" are common.

My question is: can we preprocess our data to fix this?

A specific formal problem: given k symbols and n sequences containing these symbols each of size m (symbols may be repeated), is there a invertible a way to transform these sequences into n sequences of potentially varying lengths containing a new set of w symbols (symbols may be repeated) such that:

  1. Over all sequences each new symbol is used about the same number of times

  2. Over all sequences every pair of symbols are used about the same number of times (or never used)


Any reasonably good compression algorithm should do what you want. A good compressed representation uses all symbols about equally, with little correlation between any $2$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.