3
$\begingroup$

I used StandardScaler provided by scikit-learn to scale training and validation data. Then, I fitted a neural network (CNN) model with scaled data for classification. However, in the production stage, I have to predict data in every month in the future (one-by-one). So, I use the scaler which was used in training stage to fit the new data.

My training procedure as follows:

sc = new StandardScaler()
sc.fit(train_data)
train_data = sc.transform(train_data)
build model, save(sc)

For prediction steps:

sc.fit(predict_data)
sc.transform(predict_data)
model.predict(predict_data)

There is no information about min/max in my data. I think in the prediction stage, mean and stdev of data will be changed. What should I do in this case to predict new data?

$\endgroup$
2
  • 2
    $\begingroup$ You should fit it only once on training data. The scaler saves mean and variance that you can later use for transforming new data. $\endgroup$
    – hans
    Dec 14 '17 at 9:30
  • $\begingroup$ @hans The future data might have mean value different from training data. Do you have any reference in this case? $\endgroup$
    – khant
    Dec 15 '17 at 2:16
1
$\begingroup$

You should use the same normalization approach and parameters during the model's training. In your case, you are using z-score normalization which requires two parameters namely mean $\mu$ and standard deviation $\delta$. Thus, you need to store $\mu$ and $\delta$ values in order you can scale out new data during the model's future usage / lifetime.

For example, suppose you have a dataset $D$ to be used in the training of a model $M$ using the hold-out schema. Thus, you can do:

$P, V = split(D)$

$\mu = mean(P)$

$\delta = stddev(P)$

$P_z = zscore(P; \mu, \delta)$

$V_z, = zscore(V; \mu, \delta)$

and then you can use $P_z$ to train $M$ and $V_z$ to validate if $M$ will work well on new data.

Now, consider you have new (unseen, unlabelled) data $T$ and want to use $M$ to find (predict) the labels of $T$. Thus,

$T_z = zscore(T; \mu, \delta)$

and then use the model $M$ to predict the labels of $T$ using $T_z$ as input instead of $T$ itself.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.