0
$\begingroup$

I'm creating a regular linear regression model to establish a baseline before moving on to more advanced techniques. I scaled my data as below:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train_std=pd.DataFrame(sc.fit_transform(X_train), columns=data.columns)
X_test_std=pd.DataFrame(sc.transform(X_test), columns=data.columns)

However, the variables mostly have an extreme skew (right tail), but I can't figure out how to apply a log transform on them.

Would I apply the log transform to variables in both the X_train and X_test datasets? Do I need to do this before applying the scaling? I just can't think through the right way to go about this in terms of applying predictions to the X_test set. Any ideas?

$\endgroup$
  • 1
    $\begingroup$ Scaling and then applying the log would result in errors since any values below the sample mean result in negative values post transform. Log, then scale. $\endgroup$ – Demetri Pananos Feb 11 at 18:12
0
$\begingroup$

To apply the log transform you would use numpy. Numpy as a dependency of scikit-learn and pandas so it will already be installed.

import numpy as np

X_train = np.log(X_train)
X_test = np.log(X_test)

You may also be interested in applying that transformation earlier in your pipeline before splitting data into training and test sets.

# Assumes X and y have already been defined

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X = np.log(X)

X_train, X_test, y_train, y_test = train_test_split(X, y)

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

```
| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.