0
$\begingroup$

I have a very imbalanced dataset on which I'm trying to construct a LinearSVC model with SMOTE and standardization, using a Pipeline. I had already applied SMOTE and sklearn's StandardScaler with LinearSVC, and then had constructed the same model with imblearn's make_pipeline. After having trained them both, I thought I would get the same accuracy scores in the tests, but that didn't happen.

SMOTE + StandardScaler + LinearSVC : 0.7647058823529411
SMOTE + StandardScaler + LinearSVC + make_pipeline : 0.7058823529411765

This is my code (I'll leave the imports and values for X and y in the end of the question:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=27)

pipe = make_pipeline(SMOTE(random_state=42), StandardScaler(), LinearSVC(dual=False, random_state=13))
pipe = pipe.fit(X_train, np.array(y_train))
y_pred = pipe.predict(X_test)
accuracy_1 = accuracy_score(y_test, y_pred)

# Apply SMOTE to training data and keep original test data
sm = SMOTE(random_state=27)
X_train, y_train = sm.fit_sample(X_train, np.array(y_train))

# Apply standardization after SMOTE
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

model_fitted = LinearSVC(dual=False, random_state=13).fit(X_train, np.array(y_train))
svc_pred = model_fitted.predict(X_test)
accuracy_2 = accuracy_score(y_test, svc_pred)

print(f'SMOTE + StandardScaler + LinearSVC : {accuracy_1}')
print(f'SMOTE + StandardScaler + LinearSVC + make_pipeline : {accuracy_2}')

I've read here that applying a transformer and an estimator separatly influences the model validation, because the test fold already contains information (i.e. mean and std) about the training set, since X_train was used for standardization. Is that the reason why the test metrics are different with and without make_pipeline? Am I doing something wrong in the data preprocessing? Furthermore, which would be the "correct" implementation?
Note: this also happens with other classification models I have tested, such as sklearn's RandomForestClassifier, sklearn's LogisticRegression, xgboost's XGBClassifier, etc. Bellow is a sample of my dataset. It is worth saying the class distribution in y is roughly the same as in the original data, which is approximately normally distributed.

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline
from sklearn.svm import LinearSVC
from sklearn.metrics import accuracy_score

X = pd.DataFrame([[282.54489323989515, 4, 68.16, 282.1480611960005, -6.8, 7.0403401020090985], [276.54339631417474, 4, 45.98, 342.12807470000047, -8.0, 7.152493798701296], [270.24417189647465, 4, 58.87000000000001, 343.12332366800047, -8.6, 7.137774871767122], [243.40109776254047, 4, 58.43, 309.0735966040004, -7.0, 7.064051762848258], [257.17042467369515, 4, 41.99, 276.1262631320004, -6.9, 6.311938468753469], [275.3731012246145, 7, 41.99, 330.0979976960004, -7.7, 6.7877044796869805], [265.9606513742565, 5, 51.22, 292.12117775200045, -6.9, 6.259273323565326], [326.58334774194606, 6, 103.96, 373.1096270880005, -7.5, 6.5930025859140855], [324.3768142343405, 7, 101.16, 374.0936426760005, -7.3, 6.719960972749474], [383.1075778787854, 8, 61.88000000000001, 427.14294708400075, -7.5, 7.269062418081917], [209.64125624379497, 2, 90.7, 239.080709908, -5.7, 5.592893958208465], [226.93724086966162, 3, 90.7, 253.09635997200002, -5.8, 5.802737996059506], [324.27768288769073, 6, 58.44000000000001, 372.13530359200064, -7.8, 7.227844120934621], [287.6044968433291, 5, 64.15, 374.07422332800047, -7.9, 7.139390063492067], [301.5965126238026, 5, 90.38, 370.12636044800047, -8.5, 7.351183041236545], [253.37623458851155, 7, 92.47, 325.98611209600017, -7.2, 6.186930161560662], [244.5860078879504, 6, 72.24, 309.99119747600025, -7.2, 6.725594845987346], [238.16516681807818, 6, 92.47, 292.0250844480002, -6.9, 6.042819706071707], [227.4354796800226, 5, 49.33, 281.00103388800017, -6.6, 5.568803884060376], [212.22441190958924, 4, 49.33, 247.04000624, -6.7, 5.853461533522028], [221.13625526232255, 5, 62.22, 281.99628285600016, -6.0, 5.552044152236652], [205.92518749188923, 4, 62.22, 248.035255208, -5.8, 5.520270148240644], [197.13496079132813, 3, 41.99, 232.040340588, -6.3, 5.712922288211787], [269.7450337433286, 4, 41.99, 308.0716407160004, -6.8, 5.673227140581641], [212.34602856176144, 4, 41.99, 266.0013682360002, -6.4, 5.793003591963598], [242.72369025122825, 6, 68.00999999999999, 275.0825397480004, -6.2, 5.860529551226553], [216.5158622726282, 4, 41.99, 226.110613068, -6.1, 6.271971700077697], [231.72693004306151, 5, 41.99, 260.0716407160001, -6.4, 6.3772530531690546], [225.4277056253615, 4, 54.02, 261.06688968400016, -5.3, 5.689922825396825], [252.62122300089482, 5, 41.99, 246.17321332400002, -6.1, 6.360644730824734], [249.02291466892822, 6, 41.99, 274.08729078000044, -5.8, 6.064690346597848], [250.53244991322836, 5, 91.79999999999998, 266.1167610680003, -7.1, 7.095551672050173], [239.53568970506169, 4, 65.78, 251.105862036, -6.8, 6.781502936951936], [256.83167433092837, 5, 65.78, 265.12151210000025, -6.4, 6.822023072039075], [233.23646528736165, 3, 77.81, 252.10111100400002, -5.6, 5.84349677811078], [379.1868982309576, 15, 113.58, 420.15326572800063, -5.8, 5.914399715839718], [251.78262455188363, 3, 58.64, 254.16304256400002, -6.2, 6.047892115162612], [232.97324844948372, 3, 58.64, 248.116092372, -6.6, 6.3525069249639285], [250.26923307535037, 4, 58.64, 262.1317424360002, -6.6, 5.438963752636251], [233.66639048055603, 4, 71.09, 255.100776656, -6.7, 6.572156237651242], [322.5948372520963, 6, 23.550000000000004, 320.18886338800064, -6.4, 6.457320252747259], [342.38182416082435, 7, 52.65000000000001, 351.19467704000067, -6.4, 6.904352297091798], [348.53559216058534, 6, 83.71, 365.17394159600076, -7.3, 6.91475121103896], [263.9706724272623, 5, 74.71000000000002, 316.04485458800036, -7.1, 6.2596209758574775], [273.34334207529537, 3, 23.550000000000004, 280.15756326000053, -6.7, 6.275560233877237], [279.9624427709684, 5, 23.550000000000004, 320.1091690960005, -6.9, 6.543782477633478], [242.60087095609484, 2, 61.44, 272.09833413200045, -6.8, 6.604806006160513], [249.21997165176785, 4, 61.44, 312.0499399680004, -6.9, 5.996064206404703], [375.6840430670745, 8, 61.83000000000001, 376.22497412400094, -6.6, 6.136790912698414], [356.48434271240757, 8, 57.90000000000001, 388.13107374000055, -7.8, 6.79856678421578], [296.7516255586903, 6, 68.13, 322.1555518800005, -7.4, 6.5142416255966245], [294.11516685789024, 5, 68.13, 320.13990181600053, -7.9, 7.204179028027528], [330.3981313859795, 6, 48.67, 358.1205090560005, -7.9, 6.80603791341991], [266.1537188075787, 2, 74.60000000000002, 314.0579088000003, -8.0, 6.292021111388609], [330.3981313859795, 6, 48.67, 358.1205090560005, -8.1, 6.791316292929299], [373.9773576657134, 8, 63.6, 358.21440944000085, -6.2, 6.684677804417808], [304.3119200595515, 4, 39.44, 328.10994437200037, -8.0, 6.449420691419692], [402.7000276929414, 11, 61.83000000000001, 390.2406241880009, -6.0, 6.694095905150409], [333.23162041421836, 7, 54.37, 316.2038447560008, -6.1, 6.405040869685871], [263.85065158331184, 3, 74.6, 304.07355886400035, -8.4, 6.290557231601733], [424.84863706529114, 6, 111.52, 510.13146766400047, -9.0, 6.586497894494393], [436.6704002902747, 7, 61.83000000000001, 478.1780239320007, -9.1, 7.549792955322462], [432.910874365208, 7, 72.83, 466.1780239320007, -8.1, 7.263302052170057], [314.5666155353682, 4, 111.9, 362.0790381680004, -8.2, 5.895165988566992], [338.9461414604349, 5, 108.73999999999998, 374.07903816800035, -8.2, 6.136152998556996], [411.7984310385411, 8, 68.9, 450.1467238040006, -9.1, 7.72519632156733]])
y = np.array([0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
       0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0])
$\endgroup$
3
  • $\begingroup$ what r is smote? $\endgroup$
    – develarist
    Oct 27, 2020 at 23:49
  • 1
    $\begingroup$ This looks OK to me at first glance, except that you've used a different random_state in the two SMOTEs. $\endgroup$ Oct 28, 2020 at 3:19
  • $\begingroup$ @BenReiniger thank you! I ran the model again, and the results match. Can't believe I overlooked that, but glad to know the logic is right. $\endgroup$
    – Caio Rocha
    Oct 28, 2020 at 22:07

2 Answers 2

1
$\begingroup$

You switched accuracy 1 and accuracy 2 in your print statements. Random states should all match. This answer on Stackoverflow might be helpful.

$\endgroup$
2
  • $\begingroup$ can you add a little elaboration regarding the random states and why it matters? $\endgroup$ Apr 21, 2021 at 3:49
  • $\begingroup$ also, you're right... the two accuracies are in fact switched, but I don't think that is important here, just a small oversight. perhaps move that to a comment under the OP $\endgroup$ Apr 21, 2021 at 3:51
-1
$\begingroup$

I believe the issue here is that, when you call predict on the pipeline it runs smote on test set as well. While when you do it outside the pipeline, you only run smote for the train set (which in my opinion is the correct way to do it).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.