This may well depend to an extent on how exactly you will decide which factor levels to pool.
Parameter estimates (e.g., for prediction) may even get better by pooling. Essentially, you are building a more parsimonious model. Your parameter estimates will have lower variance (because of the pooling) but higher bias, this is the ubiquitous bias-variance tradeoff. Your pooling reminds me of trees, which are certainly standard methods for prediction (with extensions, like random forests).
Inferences will probably be a little more tricky. Standard theory is not valid any more, since you are transforming your data after estimation. So don't look for $t$ tables to read off $p$ values. However, you are not filtering on low $p$ values, nor looking for "optimal cutpoints", but explicitly pooling levels with similar parameter estimates - so this could actually not be too far away from the standard tables.
If you do decide to go this way - as I said, this method could actually improve the predictive performance of your model, I recommend that you do some bootstrapping to assess how variable your parameter pooling is.