Standardization for regularized, sparse hashed logistic regression As the question states, I'm fitting large, sparse logistic regressions (with hashed interactions, a la vowpal wabbit) for a machine learning system.  The features are on different scales, and I'm a little nervous that dividing by the feature standard deviation really isn't suitable, because it seems that features that are mostly zero but are occasionally very large (I have a number of these) aren't scaled enough.  Since I have a number of hash collisions, it seems that the large values might be effectively 'swamping' the larger ones.
So, my question is: what is the advantage of scaling by the standard deviation (or root-mean-squares) instead of just dividing each feature by its maximum value, which would scale everything to the unit interval? Thanks for any help.
(note: I can't center features as that would destroy the sparsity)
 A: Its bad policity to normalize by the maximum value since the maximum is likely to change as you observe more data, thus your normalization would be inconsistent.
A: Recent Edit: Speaking directly to your comment "(note: I can't center features as that would destroy the sparsity)," actually that's not necessarily true. I've recently learned there are easy ways to have non-zero-based sparse data storage (in this case the base could be the center that you shift your data by). Something to look into...

It's not uncommon in data that exhibit sparsity (or in data that are not as smooth and well-behaved as Euclid would like) for measures such as standard deviation to be misleading or contaminated by outliers. Your intuition is correct: the variance estimator does emphasize larger values.
One solution to this problem is to consider using a robust measure of standard error. For example, consider some function of the absolute deviation—this will provide an estimate of variation or error that is far less susceptible to the extremities you see in your data.
Why not use the max? Think about stability for a moment. If you standardize by using the standard deviation and move (i.e. perturb) your data around just a bit, it's likely your standardization won't change at all because all those movements will sort of cancel out. However, if you standardize by maximum (or more accurately, $\|\cdot \|_\infty$) and again move your data around a bit, your estimate of the normalizing constant (the max) will change tremendously. Why? Because it only depends on one value (the biggest value), whereas the variance depends on all of your observations. In other words, it's sensitive, but in a different way than variance is. It's sensitive in the sense that it's unstable. Variance on the other hands has, under certain conditions, the highest level of stability of any estimator of spread, but it lacks robustness. Thus, you as a practitioner and data scientist need to strike the balances between these two kinds of estimator sensitivity—statistical efficiency and robustness. And that's why I recommend the using the absolute deviation $\|\cdot\|_1$.
In case you're wondering: Why do we standardize? Because, with several measurements that aren't naturally on the same scale, it's not easy to know what is extreme and not extreme. For example, you may have data on car speeds and airplane ground speeds per geographical region. You may see that, for a particular region, the city's car speed is perhaps 100km/hr while for an airplane, it's something like 800km/hr. Can you really compare thesd quantities and say that people in this area drive slow and fly fast? Not at all. In fact, this car speed is far above average, and the airplane ground speed is far below average. Therefore the goal of standardizing (which should truly be called normalizing) is to compare things to what is normal.
