Data normalization and sufficient statistic

Question

I was taught that when we feed our data to machine learning algorithm (e.g. SVM), we should first normalize our data.

Suppose I have a set of data $X = \{x_1,x_2,...,x_n\}$, I knew two-way of normalizing them, let $\hat{\mu}$ and $\hat{\sigma}^2$ be the sample mean and sample variance of X. I can normalize each data point by

$$ y_k = \frac{x_k-\hat{\mu}}{\hat{\sigma}} $$

I think I know this when I first learn PCA.

I can also normalize it using the minimum and maximum of the data:

$$ y_k = \frac{x-m}{M-m} $$

where $M = \max(X)$ and $m = \min(X)$. By using this normalization, I can make sure the normalized data will be in the interval $[0,1]$.

I observe a fact that:

($\hat{\mu}$,$\hat{\sigma}^2$) is sufficient statistic to a normal distribution and the projection matrix of PCA is a solution to a minimization problem that minimize $l^2$-norm.

On the other hand, $(M,m)$ is sufficient statistic to a uniform distribution.

My questions are:

1. The observation gives me an intuition that what normalization technique I should employ is depends on my belief (or the learning algorithm believes) of underlying distribution of my data. If I believe the distribution is normal distributed, I should use z-score normalization, if I believe the distribution is uniform distributed, I should use min-max normalization. Is my thought correct?

2. If I do not know the underlying distribution, how should I do data normalization?

3. If I am going to feed my data to an online learning algorithm (like winnow algorithm), are there any online data normalization technique?

Answer 1

1+2. No Clue . (I'm curious myself in fact)

2*. You might want to Simply normalize if you have large samples (Law of large numbers and all that), though it's not statistically valid. You could also try normalizing using the median. (See my link + wikipedia)

3. You save the original distribution range which you normalized by for each feature, then normalize the new (online) features using the original range. (This should be very easy to do in sci-kit learn, using fit_transform and saving the scaler). You could also update the original range every once in a while, to account for the new data. (Remember - Z-score is calculated relative to the existing samples distributions, so you can simple store them and then calc the z-score for new values by how they compare).