# Standardization for regularized, sparse hashed logistic regression

As the question states, I'm fitting large, sparse logistic regressions (with hashed interactions, a la vowpal wabbit) for a machine learning system. The features are on different scales, and I'm a little nervous that dividing by the feature standard deviation really isn't suitable, because it seems that features that are mostly zero but are occasionally very large (I have a number of these) aren't scaled enough. Since I have a number of hash collisions, it seems that the large values might be effectively 'swamping' the larger ones.

So, my question is: what is the advantage of scaling by the standard deviation (or root-mean-squares) instead of just dividing each feature by its maximum value, which would scale everything to the unit interval? Thanks for any help.

(note: I can't center features as that would destroy the sparsity)

• Can you say a bit more about your hashed interactions and "swamping the larger ones"? I can say some things about standardization in the presence of regularization, but I'd really like to know a bit more about your feature hashing and the effect of the large values first. – Matthew Drury May 22 '15 at 21:36
• Centering wouldn't necessarily destroy sparsity; that's a question of whether a given feature is used in the model, not whether it was zero to begin with. – Emre May 22 '15 at 21:58
• Tell us a little more about the nature of those features you have. – Ramalho May 22 '15 at 23:35
• @Emre I think the OP is referring to sparsely of their data matrix (for computational reasons), not sparsity in the model. – Dougal May 23 '15 at 2:36

Recent Edit: Speaking directly to your comment "(note: I can't center features as that would destroy the sparsity)," actually that's not necessarily true. I've recently learned there are easy ways to have non-zero-based sparse data storage (in this case the base could be the center that you shift your data by). Something to look into...

It's not uncommon in data that exhibit sparsity (or in data that are not as smooth and well-behaved as Euclid would like) for measures such as standard deviation to be misleading or contaminated by outliers. Your intuition is correct: the variance estimator does emphasize larger values.

One solution to this problem is to consider using a robust measure of standard error. For example, consider some function of the absolute deviation—this will provide an estimate of variation or error that is far less susceptible to the extremities you see in your data.

Why not use the max? Think about stability for a moment. If you standardize by using the standard deviation and move (i.e. perturb) your data around just a bit, it's likely your standardization won't change at all because all those movements will sort of cancel out. However, if you standardize by maximum (or more accurately, $\|\cdot \|_\infty$) and again move your data around a bit, your estimate of the normalizing constant (the max) will change tremendously. Why? Because it only depends on one value (the biggest value), whereas the variance depends on all of your observations. In other words, it's sensitive, but in a different way than variance is. It's sensitive in the sense that it's unstable. Variance on the other hands has, under certain conditions, the highest level of stability of any estimator of spread, but it lacks robustness. Thus, you as a practitioner and data scientist need to strike the balances between these two kinds of estimator sensitivity—statistical efficiency and robustness. And that's why I recommend the using the absolute deviation $\|\cdot\|_1$.

In case you're wondering: Why do we standardize? Because, with several measurements that aren't naturally on the same scale, it's not easy to know what is extreme and not extreme. For example, you may have data on car speeds and airplane ground speeds per geographical region. You may see that, for a particular region, the city's car speed is perhaps 100km/hr while for an airplane, it's something like 800km/hr. Can you really compare thesd quantities and say that people in this area drive slow and fly fast? Not at all. In fact, this car speed is far above average, and the airplane ground speed is far below average. Therefore the goal of standardizing (which should truly be called normalizing) is to compare things to what is normal.

Its bad policity to normalize by the maximum value since the maximum is likely to change as you observe more data, thus your normalization would be inconsistent.

• If you have a reasonably-sized training set, the maximum as you add more data will grow quite slowly. In cases like this where the method doesn't need the data to lie in $[-1, 1]$, it's fine if a few test points are mapped to $1.05$ or whatever. – Dougal May 23 '15 at 2:39
• If you have a reasonably sized training set... In reality, for most applications you don't, life ain't kaggle. Also, if as the op says the covariate has some instances with very large values, it suggests that there is a gap between normal instances and abnormal instances (outliers perhaps). If that's the case, and you use maximum scaling then regular values will be scaled to almost zero. That's another reason to do standardization. Also, why the downvote? – Ramalho May 23 '15 at 14:44