# How can I use scaling and log transforming together?

I'm creating a regular linear regression model to establish a baseline before moving on to more advanced techniques. I scaled my data as below:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train_std=pd.DataFrame(sc.fit_transform(X_train), columns=data.columns)
X_test_std=pd.DataFrame(sc.transform(X_test), columns=data.columns)


However, the variables mostly have an extreme skew (right tail), but I can't figure out how to apply a log transform on them.

Would I apply the log transform to variables in both the X_train and X_test datasets? Do I need to do this before applying the scaling? I just can't think through the right way to go about this in terms of applying predictions to the X_test set. Any ideas?

• Scaling and then applying the log would result in errors since any values below the sample mean result in negative values post transform. Log, then scale. – Demetri Pananos Feb 11 at 18:12

To apply the log transform you would use numpy. Numpy as a dependency of scikit-learn and pandas so it will already be installed.

import numpy as np

X_train = np.log(X_train)
X_test = np.log(X_test)


You may also be interested in applying that transformation earlier in your pipeline before splitting data into training and test sets.

# Assumes X and y have already been defined

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X = np.log(X)

X_train, X_test, y_train, y_test = train_test_split(X, y)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

$$$$
`