3
$\begingroup$

While reading a SVM tutorial, the author makes the following statement on normalization technique for processing the input data:

Normalizing data to unit vectors reduces the dimensionality of the data by one since the data is projected to the unit sphere.

I am quite lost on how to understand the dimensionality was reduced by one. Any more explanations will be greatly appreciated.

$\endgroup$
2
$\begingroup$

If you really were to normalize to unit vectors, the author would have a point: imagine every point in the plane to be "replaced" by the point on the unit circle that lies in the same direction from the origin.

Then indeed, the resulting set of points would be (at most) all points on the unit circle, which is of dimension 1 (pick any point on the unit circle as the starting point, now you can uniquely identify any other point on it by one parameter: the distance to it from the starting point along the circle)

However, typically, one doesn't actually project on the unit sphere (which is what I just described for 2D), but in a typical normalization, we simply make it so that the SD/variance is 1 (I'm ignoring the typical translation to make the mean zero here): this is not the same as projection on the unit sphere: it just brings all the data 'closer to' the unit sphere (this is an extremely inaccurate statement, but I could not immediately find one that was better suited and still related somewhat to the projection on the unit sphere idea - if relevant comments follow, I will glad to edit them in).

Disclaimer: I've not read the tutorial in question (you did not provide a link or reference), so maybe I'm answering in the wrong direction here: perhaps in the context of some example, there is a perfectly fine reason to project on the unit sphere. In that case, the explanation of my first two paragraphs should help you somewhat...

$\endgroup$
8
  • $\begingroup$ Could you elaborate on the sense in which normalizing to unit variance is not a projection onto a codimension one submanifold and how it is not a projection onto a unit sphere? I fear that either I have misunderstood what you mean, or your characterization may be incorrect. $\endgroup$ – whuber Nov 3 '11 at 17:50
  • $\begingroup$ There is some mixing up happening here: this is actually two different approaches. One is to normalize over one single data item (as meant in the tutorial) and one is to normalize to zero mean unit variance over the whole dataset (commonly called zscores). The first one clearly makes a point lie on the unit sphere (since it has length one) but the other one does not necessarily. E.g. the 1D points (2, 3, 4) (representing a data set $X \subset \mathbb{R}^1$ here!) are not 0D after taking their zscores which would make them (-1.2, 0, 1.2). $\endgroup$ – bayerj Nov 3 '11 at 22:39
  • $\begingroup$ Furthermore, there are some applications in machine learning where this is typically done. E.g. for image processing, this helps to reduce lighting effects etc. $\endgroup$ – bayerj Nov 3 '11 at 22:41
  • $\begingroup$ @whuber: bayer is exactly right. The normalization I have encountered myself in my projects is all "zero mean unit variance" (and I can see how this is remotely related to "projection on the unit sphere", though it is clearly no such thing). I still cannot get my head around why you would want to project on the unit sphere - it seems like a waste of valuable information to me, and even as such surely is a dimension reduction. But like I said: somebody point me to the tutorial in question and I will stand corrected. And besides: bayer does confirm it is used in some applications... $\endgroup$ – Nick Sabbe Nov 4 '11 at 8:07
  • $\begingroup$ @Nick Standardizing to zero mean unit variance over the whole dataset actually reduces the dimension by two: it projects onto a (not the) sphere within a codimension-one subspace (namely, that orthogonal to the vector $\mathbf{1}$). So, technically, "the" and "unit" may be incorrect characterizations in this case, but the basic conclusion stands: that the dimensionality of the data is reduced. $\endgroup$ – whuber Nov 4 '11 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.