# How can I scale new data which were not used in training/validating?

I used StandardScaler provided by scikit-learn to scale training and validation data. Then, I fitted a neural network (CNN) model with scaled data for classification. However, in the production stage, I have to predict data in every month in the future (one-by-one). So, I use the scaler which was used in training stage to fit the new data.

My training procedure as follows:

sc = new StandardScaler()
sc.fit(train_data)
train_data = sc.transform(train_data)
build model, save(sc)


For prediction steps:

sc.fit(predict_data)
sc.transform(predict_data)
model.predict(predict_data)


There is no information about min/max in my data. I think in the prediction stage, mean and stdev of data will be changed. What should I do in this case to predict new data?

• You should fit it only once on training data. The scaler saves mean and variance that you can later use for transforming new data.
– hans
Dec 14 '17 at 9:30
• @hans The future data might have mean value different from training data. Do you have any reference in this case? Dec 15 '17 at 2:16

You should use the same normalization approach and parameters during the model's training. In your case, you are using z-score normalization which requires two parameters namely mean $$\mu$$ and standard deviation $$\delta$$. Thus, you need to store $$\mu$$ and $$\delta$$ values in order you can scale out new data during the model's future usage / lifetime.

For example, suppose you have a dataset $$D$$ to be used in the training of a model $$M$$ using the hold-out schema. Thus, you can do:

$$P, V = split(D)$$

$$\mu = mean(P)$$

$$\delta = stddev(P)$$

$$P_z = zscore(P; \mu, \delta)$$

$$V_z, = zscore(V; \mu, \delta)$$

and then you can use $$P_z$$ to train $$M$$ and $$V_z$$ to validate if $$M$$ will work well on new data.

Now, consider you have new (unseen, unlabelled) data $$T$$ and want to use $$M$$ to find (predict) the labels of $$T$$. Thus,

$$T_z = zscore(T; \mu, \delta)$$

and then use the model $$M$$ to predict the labels of $$T$$ using $$T_z$$ as input instead of $$T$$ itself.