2
$\begingroup$

Suppose I have some input data X with shape [n,m]. I also have some target data y with shape [s,p].

I want to train a model with some train data and then compare the results on the test data, as usual. I came across the idea of normalizing the data. However, I don't understand (although the documentation is pretty good here) whether I should use one min max scaler for the input and one for the ouput or just one? Also, should I apply it on the training set only right? Not training + test?

SCALER FOR X_TRAIN, SCALER FOR Y_TRAIN

# divide into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X,y, shuffle = False, tests_size = 0.33)
# create scalers
scalerX = MinMaxScaler(feature_range = (0,1))
scalery = MinMaxScaler(feature_range = (0,1))
# fit and transform
X_train = scalerX.fit_transform(X_train)
y_y_train = scalery.fit_transform(y_train)
X_test = scalerX.transform(X_test)
y_test = scalery.transform(y_test)
# then once I will have the predictions
y_pred = scalery.inverse_transform(y_pred)

SCALER FOR X , SCALER FOR Y

scalerX = MinMaxScaler(feature_range = (0,1))
scaley = MinMaxScaler(feature_range = (0,1))
X = scalerX.fit_transform(X)
y = scalery.fit_transform(y)
# training and testing
X_train, X_test, y_train, y_test = train_test_split(X,y, shuffle = False, test_size = 0.33)
# then once I will have the predictions
y_pred = scalery.inverse_transform(y_pred)

ONE SCALER FOR XY MATRIX

Xy = np.hstack((X, y))
scaler = MinMaxScaler(feature_range = (0,1))
Xy = scaler.fit_transform(Xy)
# then separate training and test
X_train, X_test, y_train, y_test = train_test_split(Xy,shuffle = False, test_size = 0.33)
# once I will have the predictions
y_pred = scaler.transform(y_pred)

Which one of these three options is the correct one? Or are they all wrong?

$\endgroup$
1
  • $\begingroup$ There's an error in the last part. train_test_split(Xy,shuffle = False, test_size = 0.33) returns two arrays, not four. Perhaps you should split Xy to X and y? $\endgroup$
    – mavavilj
    Nov 26, 2018 at 12:47

2 Answers 2

2
$\begingroup$

Use a single scaler, fit on the train set. It's best to pretend that you are in production, and don't actually have the test dataset. If you fit a separate scaler, you are using information you shouldn't have.

$\endgroup$
10
  • $\begingroup$ Exactly what I thought! In reality one wouldn't have the test set. Just to confirm, should I use it ONLY ONE SCALER on `np.hstack((X_train, y_train)) or should I use one scaler for X_train and one for y_train? $\endgroup$ Oct 31, 2017 at 12:41
  • $\begingroup$ only one scaler. If you are worried that some values might be e.g. over 1 in the test set, and the model depends on the values being lower than 1, set the limits of the scaler perhaps at 0.05, 0.95. $\endgroup$ Oct 31, 2017 at 13:16
  • $\begingroup$ it raises an error for the shape when I try converting the predictions.. $\endgroup$ Oct 31, 2017 at 13:46
  • $\begingroup$ can you show the error? $\endgroup$ Oct 31, 2017 at 14:11
  • $\begingroup$ ValueError: operands could not be broadcast together with shapes (2,2) (5,) (2,2) $\endgroup$ Oct 31, 2017 at 14:17
1
$\begingroup$

I know this is late but maybe my answer will help some one else looking for the answer like I did.

To make this short, It does not matter if you use 2 individual scalers for X and y or if you scale the whole dataset(X and y) with 1 scaler.

Both will be scaled the same to the exact same vale.

So if you want to easily inverse transform only the target value, then I would recommend using 2 scalers, one for X and one for y

And like some comments already mentioned, your scaler should be fitted using only the training data.

Therefore, I would implement it like SCALER FOR X_TRAIN, SCALER FOR Y_TRAIN

If you use ONE SCALER FOR XY MATRIX, then you have to include some X values to re-scale, otherwise it won't work

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.