0
$\begingroup$

I have a very imbalanced dataset on which I'm trying to construct a LinearSVC model with SMOTE and standardization, using a Pipeline. I had already applied SMOTE and sklearn's StandardScaler with LinearSVC, and then had constructed the same model with imblearn's make_pipeline. After having trained them both, I thought I would get the same accuracy scores in the tests, but that didn't happen.

SMOTE + StandardScaler + LinearSVC : 0.7647058823529411
SMOTE + StandardScaler + LinearSVC + make_pipeline : 0.7058823529411765

This is my code (I'll leave the imports and values for X and y in the end of the question:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=27)

pipe = make_pipeline(SMOTE(random_state=42), StandardScaler(), LinearSVC(dual=False, random_state=13))
pipe = pipe.fit(X_train, np.array(y_train))
y_pred = pipe.predict(X_test)
accuracy_1 = accuracy_score(y_test, y_pred)

# Apply SMOTE to training data and keep original test data
sm = SMOTE(random_state=27)
X_train, y_train = sm.fit_sample(X_train, np.array(y_train))

# Apply standardization after SMOTE
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

model_fitted = LinearSVC(dual=False, random_state=13).fit(X_train, np.array(y_train))
svc_pred = model_fitted.predict(X_test)
accuracy_2 = accuracy_score(y_test, svc_pred)

print(f'SMOTE + StandardScaler + LinearSVC : {accuracy_1}')
print(f'SMOTE + StandardScaler + LinearSVC + make_pipeline : {accuracy_2}')

I've read here that applying a transformer and an estimator separatly influences the model validation, because the test fold already contains information (i.e. mean and std) about the training set, since X_train was used for standardization. Is that the reason why the test metrics are different with and without make_pipeline? Am I doing something wrong in the data preprocessing? Furthermore, which would be the "correct" implementation?
Note: this also happens with other classification models I have tested, such as sklearn's RandomForestClassifier, sklearn's LogisticRegression, xgboost's XGBClassifier, etc. Bellow is a sample of my dataset. It is worth saying the class distribution in y is roughly the same as in the original data, which is approximately normally distributed.

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline
from sklearn.svm import LinearSVC
from sklearn.metrics import accuracy_score

X = pd.DataFrame([[282.54489323989515, 4, 68.16, 282.1480611960005, -6.8, 7.0403401020090985], [276.54339631417474, 4, 45.98, 342.12807470000047, -8.0, 7.152493798701296], [270.24417189647465, 4, 58.87000000000001, 343.12332366800047, -8.6, 7.137774871767122], [243.40109776254047, 4, 58.43, 309.0735966040004, -7.0, 7.064051762848258], [257.17042467369515, 4, 41.99, 276.1262631320004, -6.9, 6.311938468753469], [275.3731012246145, 7, 41.99, 330.0979976960004, -7.7, 6.7877044796869805], [265.9606513742565, 5, 51.22, 292.12117775200045, -6.9, 6.259273323565326], [326.58334774194606, 6, 103.96, 373.1096270880005, -7.5, 6.5930025859140855], [324.3768142343405, 7, 101.16, 374.0936426760005, -7.3, 6.719960972749474], [383.1075778787854, 8, 61.88000000000001, 427.14294708400075, -7.5, 7.269062418081917], [209.64125624379497, 2, 90.7, 239.080709908, -5.7, 5.592893958208465], [226.93724086966162, 3, 90.7, 253.09635997200002, -5.8, 5.802737996059506], [324.27768288769073, 6, 58.44000000000001, 372.13530359200064, -7.8, 7.227844120934621], [287.6044968433291, 5, 64.15, 374.07422332800047, -7.9, 7.139390063492067], [301.5965126238026, 5, 90.38, 370.12636044800047, -8.5, 7.351183041236545], [253.37623458851155, 7, 92.47, 325.98611209600017, -7.2, 6.186930161560662], [244.5860078879504, 6, 72.24, 309.99119747600025, -7.2, 6.725594845987346], [238.16516681807818, 6, 92.47, 292.0250844480002, -6.9, 6.042819706071707], [227.4354796800226, 5, 49.33, 281.00103388800017, -6.6, 5.568803884060376], [212.22441190958924, 4, 49.33, 247.04000624, -6.7, 5.853461533522028], [221.13625526232255, 5, 62.22, 281.99628285600016, -6.0, 5.552044152236652], [205.92518749188923, 4, 62.22, 248.035255208, -5.8, 5.520270148240644], [197.13496079132813, 3, 41.99, 232.040340588, -6.3, 5.712922288211787], [269.7450337433286, 4, 41.99, 308.0716407160004, -6.8, 5.673227140581641], [212.34602856176144, 4, 41.99, 266.0013682360002, -6.4, 5.793003591963598], [242.72369025122825, 6, 68.00999999999999, 275.0825397480004, -6.2, 5.860529551226553], [216.5158622726282, 4, 41.99, 226.110613068, -6.1, 6.271971700077697], [231.72693004306151, 5, 41.99, 260.0716407160001, -6.4, 6.3772530531690546], [225.4277056253615, 4, 54.02, 261.06688968400016, -5.3, 5.689922825396825], [252.62122300089482, 5, 41.99, 246.17321332400002, -6.1, 6.360644730824734], [249.02291466892822, 6, 41.99, 274.08729078000044, -5.8, 6.064690346597848], [250.53244991322836, 5, 91.79999999999998, 266.1167610680003, -7.1, 7.095551672050173], [239.53568970506169, 4, 65.78, 251.105862036, -6.8, 6.781502936951936], [256.83167433092837, 5, 65.78, 265.12151210000025, -6.4, 6.822023072039075], [233.23646528736165, 3, 77.81, 252.10111100400002, -5.6, 5.84349677811078], [379.1868982309576, 15, 113.58, 420.15326572800063, -5.8, 5.914399715839718], [251.78262455188363, 3, 58.64, 254.16304256400002, -6.2, 6.047892115162612], [232.97324844948372, 3, 58.64, 248.116092372, -6.6, 6.3525069249639285], [250.26923307535037, 4, 58.64, 262.1317424360002, -6.6, 5.438963752636251], [233.66639048055603, 4, 71.09, 255.100776656, -6.7, 6.572156237651242], [322.5948372520963, 6, 23.550000000000004, 320.18886338800064, -6.4, 6.457320252747259], [342.38182416082435, 7, 52.65000000000001, 351.19467704000067, -6.4, 6.904352297091798], [348.53559216058534, 6, 83.71, 365.17394159600076, -7.3, 6.91475121103896], [263.9706724272623, 5, 74.71000000000002, 316.04485458800036, -7.1, 6.2596209758574775], [273.34334207529537, 3, 23.550000000000004, 280.15756326000053, -6.7, 6.275560233877237], [279.9624427709684, 5, 23.550000000000004, 320.1091690960005, -6.9, 6.543782477633478], [242.60087095609484, 2, 61.44, 272.09833413200045, -6.8, 6.604806006160513], [249.21997165176785, 4, 61.44, 312.0499399680004, -6.9, 5.996064206404703], [375.6840430670745, 8, 61.83000000000001, 376.22497412400094, -6.6, 6.136790912698414], [356.48434271240757, 8, 57.90000000000001, 388.13107374000055, -7.8, 6.79856678421578], [296.7516255586903, 6, 68.13, 322.1555518800005, -7.4, 6.5142416255966245], [294.11516685789024, 5, 68.13, 320.13990181600053, -7.9, 7.204179028027528], [330.3981313859795, 6, 48.67, 358.1205090560005, -7.9, 6.80603791341991], [266.1537188075787, 2, 74.60000000000002, 314.0579088000003, -8.0, 6.292021111388609], [330.3981313859795, 6, 48.67, 358.1205090560005, -8.1, 6.791316292929299], [373.9773576657134, 8, 63.6, 358.21440944000085, -6.2, 6.684677804417808], [304.3119200595515, 4, 39.44, 328.10994437200037, -8.0, 6.449420691419692], [402.7000276929414, 11, 61.83000000000001, 390.2406241880009, -6.0, 6.694095905150409], [333.23162041421836, 7, 54.37, 316.2038447560008, -6.1, 6.405040869685871], [263.85065158331184, 3, 74.6, 304.07355886400035, -8.4, 6.290557231601733], [424.84863706529114, 6, 111.52, 510.13146766400047, -9.0, 6.586497894494393], [436.6704002902747, 7, 61.83000000000001, 478.1780239320007, -9.1, 7.549792955322462], [432.910874365208, 7, 72.83, 466.1780239320007, -8.1, 7.263302052170057], [314.5666155353682, 4, 111.9, 362.0790381680004, -8.2, 5.895165988566992], [338.9461414604349, 5, 108.73999999999998, 374.07903816800035, -8.2, 6.136152998556996], [411.7984310385411, 8, 68.9, 450.1467238040006, -9.1, 7.72519632156733]])
y = np.array([0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
       0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0])
$\endgroup$
3
  • $\begingroup$ what r is smote? $\endgroup$ – develarist Oct 27 '20 at 23:49
  • 1
    $\begingroup$ This looks OK to me at first glance, except that you've used a different random_state in the two SMOTEs. $\endgroup$ – Ben Reiniger Oct 28 '20 at 3:19
  • $\begingroup$ @BenReiniger thank you! I ran the model again, and the results match. Can't believe I overlooked that, but glad to know the logic is right. $\endgroup$ – Caio Rocha Oct 28 '20 at 22:07
1
$\begingroup$

You switched accuracy 1 and accuracy 2 in your print statements. Random states should all match. This answer on Stackoverflow might be helpful.

$\endgroup$
2
  • $\begingroup$ can you add a little elaboration regarding the random states and why it matters? $\endgroup$ – Kevin Glynn Apr 21 at 3:49
  • $\begingroup$ also, you're right... the two accuracies are in fact switched, but I don't think that is important here, just a small oversight. perhaps move that to a comment under the OP $\endgroup$ – Kevin Glynn Apr 21 at 3:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.