Data normalization and sufficient statistic

I was taught that when we feed our data to machine learning algorithm (e.g. SVM), we should first normalize our data.

Suppose I have a set of data $X = \{x_1,x_2,...,x_n\}$, I knew two-way of normalizing them, let $\hat{\mu}$ and $\hat{\sigma}^2$ be the sample mean and sample variance of X. I can normalize each data point by

$$y_k = \frac{x_k-\hat{\mu}}{\hat{\sigma}}$$

I think I know this when I first learn PCA.

I can also normalize it using the minimum and maximum of the data:

$$y_k = \frac{x-m}{M-m}$$

where $M = \max(X)$ and $m = \min(X)$. By using this normalization, I can make sure the normalized data will be in the interval $[0,1]$.

I observe a fact that:

($\hat{\mu}$,$\hat{\sigma}^2$) is sufficient statistic to a normal distribution and the projection matrix of PCA is a solution to a minimization problem that minimize $l^2$-norm.

On the other hand, $(M,m)$ is sufficient statistic to a uniform distribution.

My questions are:

1. The observation gives me an intuition that what normalization technique I should employ is depends on my belief (or the learning algorithm believes) of underlying distribution of my data. If I believe the distribution is normal distributed, I should use z-score normalization, if I believe the distribution is uniform distributed, I should use min-max normalization. Is my thought correct?

2. If I do not know the underlying distribution, how should I do data normalization?

3. If I am going to feed my data to an online learning algorithm (like winnow algorithm), are there any online data normalization technique?

• I came across your question as I was looking for references for bounded continuous distributions, normalized by a "range", that weren't necessarily uniform. Hopefully this is still marginally helpful: en.wikipedia.org/wiki/Kumaraswamy_distribution – user106121 Feb 23 '16 at 3:09