-2
$\begingroup$

I am using the StandardScaler of Scikit-learn to scale my data. When plotting the data and comparing scaled with unscaled I get different results in how the plots look like. I expect that the plot of the unscaled data looks like that one of the scaled one but instead they look really different. Why is that the case? I thought during scaling the relations between the data are still the same.

Unscaled data

$value_1 = 1011.96875, value_2 = 1011.87500, value_3 = 1011.78125 ... value_{1000} = 937.43750$

enter image description here

Scaled data

$value_1 = -0.468095, value_2 = -0.258494, value_3 = -0.191027 ... value_{1000} = -0.028573$

enter image description here

Code for scaling

train_x, test_x, train_y, test_y = get_data_unscaled()
test_x = pd.DataFrame(test_x)
test_y = pd.DataFrame(test_y)
train_x = pd.DataFrame(train_x)
train_y = pd.DataFrame(train_y)
scalerX = StandardScaler().fit(train_x)
scalerY = StandardScaler().fit(train_y)
train_x = scalerX.transform(train_x)
train_y = scalerY.transform(train_y)
test_x = scalerX.transform(test_x)
test_y = scalerY.transform(test_y)

where train_x, test_x, train_y, test_y is the unscaled data.

$\endgroup$
11
  • 1
    $\begingroup$ The labels on the horizontal axes are completely different in the two plots. Are you sure that both plots these show the same data? $\endgroup$
    – Sycorax
    Commented Nov 9, 2022 at 19:15
  • 1
    $\begingroup$ Welcome to CV, Max Hager. Can you tell us what mathematical function your are referring to with the word "scaled"? Often time "scaled" means multiplied by some number (e.g., the reciprocal of the standard deviation is a common scaling factor). However, if you are using "scaled" to mean some mathematically more complex transformation, it will help your audience, including @Sycorax , to understand your question better. (You can click "edit" in the lower left of your question to add that information.) $\endgroup$
    – Alexis
    Commented Nov 9, 2022 at 19:18
  • 3
    $\begingroup$ Can you plot scaled vs unscaled? Should be linear on your story. $\endgroup$
    – Nick Cox
    Commented Nov 9, 2022 at 19:24
  • 1
    $\begingroup$ If I had to guess what happened, I think that the second plot is produced from 2 transformations: (1) first-differences and (2) standard scalar. The first plot shows a drop off on the left and "leveling off" to the right, and the second plot shows steep values on the left and values near zero on the right. $\endgroup$
    – Sycorax
    Commented Nov 9, 2022 at 19:24
  • 2
    $\begingroup$ Nick Cox's comment is exactly correct: if the scaled vs unscaled data plot is not a straight line, then you applied some additional transformations. The plots don't show the unscaled data plotted against the scaled data, they show the scaled data plotted against a sequence of integers and the unscaled data plotted against a series of strings. The code is not completely helpful because it doesn't show how you made the plots. First-differences compute 1-step changes: $\Delta x = x_{t+1} - x_t$. $\endgroup$
    – Sycorax
    Commented Nov 9, 2022 at 19:54

1 Answer 1

4
$\begingroup$

Nick Cox's comment is exactly correct. The sklearn StandardScalar computes new variables that have the mean $\mu$ subtracted and are divided by the standard deviation $\sigma > 0$: $$\begin{align} \tilde x &= \frac{x - \mu}{\sigma} \\ &= \sigma^{-1}x - \sigma^{-1}\mu \end{align} $$ Recall that the equation of a line is $y=ax+b$, so we have $a = \sigma^{-1}$ and $b =- \sigma^{-1}\mu$. Thus, $\tilde x$ is a linear function of $x$, so a plot of $(x,\tilde x)$ should form a straight line.

If the plot of scaled data $\tilde x$ vs unscaled data $x$ is not a straight line, then you've made a mistake somewhere in your code (perhaps you applied some additional transformations, or are comparing two different series of data, or you plotted the wrong data, or you used the wrong plotting options).

If you're using Jupyter Notebooks, I think the most plausible source of error is that you've run the cells in an unexpected order.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.