I've noticed that many pattern predictors initially use the solution of "always pick the most common thing" and sometimes struggle to ever get out of local holes like that for a while.
A simple example is text: at the character level "e" is very very common while at the word level words like "the" and "a" are common.
My question is: can we preprocess our data to fix this?
A specific formal problem: given k symbols and n sequences containing these symbols each of size m (symbols may be repeated), is there a invertible a way to transform these sequences into n sequences of potentially varying lengths containing a new set of w symbols (symbols may be repeated) such that:
Over all sequences each new symbol is used about the same number of times
Over all sequences every pair of symbols are used about the same number of times (or never used)