1
$\begingroup$

I am dealing with panel data and want to scale it in order to use it for some ML models:

id year A B C
1 2000 3,539,101 265.152 .0683649
1 2001 3,539.101 2,485.833 .0683649
1 2002 3,539.101 2,939.903 .0688288
1 2003 3,733.545 3,021.591 -.0257413
2 2000 3,960.184 9,418.228 .9781774
2 2001 3,960.184 9,418.228 .4855057
2 2002 3,960.184 9,880.249 .049056
2 2003 3,960.184 1,287.206 .2310434
3 2000 4,724.285 1,287.206 -.0373083
3 2001 4,724.285 1,582.817 .1202868
3 2002 4,724.285 1,279.348 -.1824576
3 2003 4,724.285 1,213.678 -.0513311

However, I'm not sure if I should scale per group (ID) or not. I'm using the following code:

features = df.columns[2:5].tolist()
df[features]=sklearn.preprocessing.minmax_scale(df[features], feature_range=(0, 1), axis=0, copy=False)

Or should I do something like this instead:

from sklearn.preprocessing import minmax_scale()
df[features]=df.groupby("id")[features].transform(lambda x: minmax_scale(x.astype(float)))


Or should I scale by year?

df[features]=df.groupby("year")[features].transform(lambda x: minmax_scale(x.astype(float)))

$\endgroup$

1 Answer 1

1
$\begingroup$

It really depends on what you want to do. The scaling in this case affects your data and the information it contains.

To be concrete, let's refer to the following example. It is a stylized version of your data. The dataset I am using is as follows:

    id  year   A            B
0    1  2000   0     0.000000
1    1  2001   1     1.000000
2    1  2002   2     2.000000
3    1  2003   3     3.000000
4    1  2004   4     4.000000
5    1  2005   5     5.000000
6    1  2006   6     6.000000
7    1  2007   7     7.000000
8    1  2008   8     8.000000
9    1  2009   9     9.000000
10   2  2000  10  1100.000000
11   2  2001   9  1050.000000
12   2  2002   8  1033.333333
13   2  2003   7  1025.000000
14   2  2004   6  1020.000000
15   2  2005   5  1016.666667
16   2  2006   4  1014.285714
17   2  2007   3  1012.500000
18   2  2008   2  1011.111111
19   2  2009   1  1010.000000

We have two groups, 10 years for each and two features: A and B. This data is used to produce the following graph.

enter image description here

On the x-axis you read the years; on the y-axis you read the values of either feature A (first column) or feature B (second column). Colors represent the two IDs (violet is ID=1; yellow is ID=2).

The first row is the original data, not scaled:

  • feature A is increasing for ID=1 and it is decreasing for ID=2; they show the same scale.

  • feature B shows different dynamics for the two groups; still it is increasing for group 1 and decreasing for group 2. They are on different scales.

Consider row by row:

  • the first row is a simple plot of the data. You can reconcile it with the description above. Note that since B is on different scales for the two groups, you can not really notice the groups' dynamics across years.

  • the second row is a simple scaling obtained with your first proposal: df[features]=sklearn.preprocessing.minmax_scale(df[features], feature_range=(0, 1), axis=0, copy=False). Here, you can notice that the graphs are identical to those in the first rows. The only noticeable difference is the scale of the y-axis being in 0-1. This transformation preserves the original information both with respect to years and to groups.

  • the third row is the group-scaling (your second proposal: df[features]=df.groupby("id")[features].transform(lambda x: minmax_scale(x.astype(float)))). Here things change a lot. Basically, this transformation removes group specific scales, preserving year dynamics. Here you lose information relative to group different scales which may be relevant. E.g. is it important that group 2 has a value of the B feature that is 1000 times that of group 1?

  • the last one is obtained scaling by year (df[features]=df.groupby("year")[features].transform(lambda x: minmax_scale(x.astype(float)))). Here you lose information on year dynamics, but keep information of different scales. E.g. you are not longer able to see increasing/decreasing dynamics across years, but you see that the value of the A feature is bigger for group 1 till some point than the reverse is true. For feature B, instead, you simply loose increasing and decreasing patterns.

Which one is best?

This depends on the data, prior knowledge and goal of the analysis. In general, if you want simply to take all the features to comparable magnitude I would go for the "Vanilla feature scaling", which preserve basic information, but makes so that all the features are in the same range. This is also what ML algorithms require, typically.

However, you may be interested in yearly dynamics. In this case, scaling within groups could make the problem easier to solve.

Or it may happen that you know that your data is such that measurements across years use different scales, but there is no dynamic component you want to capture; then, scale within years.

Keep in mind that within-years and within-groups scaling affect the final information you are using.

One last consideration is: if you are unsure which one you should use and you are dealing with a purely prediction problem where you can check to goodness of final results, you can just make this decision part of the algorithm's tuning: try all of them and select the best one via cross-validation, for example.

Code used for the example

from sklearn.preprocessing import minmax_scale
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt


A1 = np.arange(10)
B1 = np.arange(10)
ID1 = np.repeat(1, 10)
YEAR1 = np.arange(2000, 2010)

A2 = 10 - np.arange(10)
B2 = 1000 + 1 / np.arange(0.01, 0.11, 0.01)
ID2 = np.repeat(2, 10)
YEAR2 = np.arange(2000, 2010)


A = np.concatenate([A1, A2])
B = np.concatenate([B1, B2])
ID = np.concatenate([ID1, ID2])
YEAR = np.concatenate([YEAR1, YEAR2])

dat = pd.DataFrame({"id": ID, "year": YEAR, "A": A, "B": B})
features = ["A", "B"]


fig, ax = plt.subplots(nrows=4, ncols=2, sharex=True, figsize=(12, 12))

# Plain fig
ax[0, 0].scatter(dat.year, dat.A, c=dat.id)
ax[0, 1].scatter(dat.year, dat.B, c=dat.id)
ax[0, 0].set_title("No scale (A)")
ax[0, 1].set_title("No scale (B)")

# Scale entire feature
df = dat.copy()
df[features] = minmax_scale(df[features], feature_range=(0, 1), axis=0, copy=False)
ax[1, 0].scatter(df.year, df.A, c=df.id)
ax[1, 1].scatter(df.year, df.B, c=df.id)
ax[1, 0].set_title("Vanilla feature scaling (A)")
ax[1, 1].set_title("Vanilla feature scaling (B)")

# Scale by group (id)
df = dat.copy()
df[features] = df.groupby("id")[features].transform(
    lambda x: minmax_scale(x.astype(float))
)
ax[2, 0].scatter(df.year, df.A, c=df.id)
ax[2, 1].scatter(df.year, df.B, c=df.id)
ax[2, 0].set_title("By-ID feature scaling (A)")
ax[2, 1].set_title("By-ID feature scaling (B)")


# Scale by year
df = dat.copy()
df[features] = df.groupby("year")[features].transform(
    lambda x: minmax_scale(x.astype(float))
)
ax[3, 0].scatter(df.year, df.A, c=df.id)
ax[3, 1].scatter(df.year, df.B, c=df.id)
ax[3, 0].set_title("By-YEAR feature scaling (A)")
ax[3, 1].set_title("By-YEAR feature scaling (B)")

plt.show()
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.