How can I use scaling and log transforming together?

I'm creating a regular linear regression model to establish a baseline before moving on to more advanced techniques. I scaled my data as below:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train_std=pd.DataFrame(sc.fit_transform(X_train), columns=data.columns)
X_test_std=pd.DataFrame(sc.transform(X_test), columns=data.columns)


However, the variables mostly have an extreme skew (right tail), but I can't figure out how to apply a log transform on them.

Would I apply the log transform to variables in both the X_train and X_test datasets? Do I need to do this before applying the scaling? I just can't think through the right way to go about this in terms of applying predictions to the X_test set. Any ideas?

• Scaling and then applying the log would result in errors since any values below the sample mean result in negative values post transform. Log, then scale. – Demetri Pananos Feb 11 '20 at 18:12

To apply the log transform you would use numpy. Numpy as a dependency of scikit-learn and pandas so it will already be installed.

import numpy as np

X_train = np.log(X_train)
X_test = np.log(X_test)


You may also be interested in applying that transformation earlier in your pipeline before splitting data into training and test sets.

# Assumes X and y have already been defined

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X = np.log(X)

X_train, X_test, y_train, y_test = train_test_split(X, y)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

$$$$


You can form a pipeline and apply standard scaling and log transformation subsequently. In this way, you can just train your pipelined regressor on the train data and then use it on the test data. For every input, the pipelined regressor will standardize and log transform the input before making the prediction.

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import FunctionTransformer
from imblearn.pipeline import Pipeline

def log_transform(x):
print(x)
return np.log(x + 1)

scaler = StandardScaler()
transformer = FunctionTransformer(log_transform)
pipe = Pipeline(steps=[('scaler', scaler), ('transformer', transformer), ('regressor', your_regressor)], memory='sklearn_tmp_memory')

pipe.fit(X_train, y_train)
pipe.score(X_test, y_test)
`
• You can't do log on negative data.. – Ferus Feb 6 at 11:22