I am learning LinearRegression (specifically in
sklearn; Python's SciKit library) We are making models, fitting them with training datasets, then scoring them against datasets:
model = LinearRegression() model.fit(X_train, y_train) score_on_train = model.score(X_train, y_train) score_on_test = model.score(X_test, y_test)
My class materials materials say:
the model should always perform better on the training set than the testing set. This because the model was trained on the training data and not on the testing data. Intuitively, the model should perform better on data that it has seen before versus data it has not seen.
But this is not true for my datasets; the model doesn't perform better on training data;
model.score(...) on the training dataset was lower than scoring the test dataset!
score_on_train < score_on_test
But I am tempted by this "Intuitively..." explanation.
Is it always true that a model will perform better on its training data than some test data ? Why or why not ? Maybe the text I quoted is trying to describe a different phenomenon.
So far, responses suggest the model should perform better on training data most of the time. But I tried this suggestion:
"Try different train/test splits and see if the problem persists." when I run 1000 trials of 1000
make_regression simulated data : the training data scores higher in only ~50% of cases; hardly most of the time.
Am I doing something wrong? How can I avoid "information leaking"?
from sklearn.preprocessing import StandardScaler from sklearn.model_selection import train_test_split from sklearn.datasets import make_regression from sklearn.metrics import r2_score, mean_squared_error import math results= #~100 trials for i in range(1,1000): #In each trial, generate 1000 random observations X, y = make_regression(n_features=1, n_samples=1000, noise = 4, random_state=i) y=y.reshape(-1, 1) #split observations into training and testing X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=i, train_size=0.8)#42) #Scale... (am I doing this properly?) X_scaler = StandardScaler().fit(X_train) y_scaler = StandardScaler().fit(y_train) X_train_scaled = X_scaler.transform(X_train) X_test_scaled = X_scaler.transform(X_test) y_train_scaled = y_scaler.transform(y_train) y_test_scaled = y_scaler.transform(y_test) mdl = LinearRegression() #Train the model to the training data mdl.fit(X_train_scaled, y_train_scaled) #But score the model on the training data, *and the test data* results.append(( #mdl.score does R-squared coefficient, so this code is equivalent: r2_score(y_train_scaled, mdl.predict(X_train_scaled)), r2_score(y_test_scaled, mdl.predict(X_test_scaled)), # mdl.score(X_train_scaled, y_train_scaled), # mdl.score(X_test_scaled, y_test_scaled) # https://stackoverflow.com/a/18623635/1175496 math.sqrt(mean_squared_error(y_train_scaled, mdl.predict(X_train_scaled))), math.sqrt(mean_squared_error(y_test_scaled, mdl.predict(X_test_scaled))) )) train_vs_test_df = pd.DataFrame(results, columns=('r2__train', 'r2__test', 'rmse__train', 'rmse__test')) # Count how frequently the winner is the model's score on training data set train_vs_test_df['r2__winner_is_train'] = train_vs_test_df['r2__train'] > train_vs_test_df['r2__test'] train_vs_test_df['rmse__winner_is_train'] = train_vs_test_df['rmse__train'] > train_vs_test_df['rmse__test'] train_vs_test_df.head(10)
And when I check how many times the training data scored better:(497, 505)
( train_vs_test_df['r2__winner_is_train'].sum(), train_vs_test_df['rmse__winner_is_train'].sum() )
... training data scores a higher R-squared score in only
And the training data scores a higher RMSE-score in only
507 cases! (meaning it's only better in 493 cases). In other words, roughly half! (This is very different than "always" / "almost always" which I am led to believe)
When I change the above parameters, (like changing what amount is used as training data vs amount used as test data... or changing the sample size... or changing the random_state... the test data performs better only about half the time?