0
$\begingroup$

I am new to Machine Learning, as part of my learning, I created a Linear Regression Model using two attributes for Boston Housing Price dataset. I am having doubts calculating : MSE and R-Square. So, if I train & split the data and calculate the R-square for Test or Training data and then if I use the k-fold cross validation, there is a significant difference in the average value. (0.11 vs 0.63 / 0.57)

Now, I understand that this is not the perfect model. But as I am beginner, I just wanted to understand if this kind of difference between values is normal and based on average cross validation score, we say this model is not the best fit irrspective of test/train r-square value? How would you interpret this difference?

import numpy as np
import pandas as pd
from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score
from sklearn.model_selection import cross_val_score

boston = load_boston()
df = pd.DataFrame(boston.data)
df.columns = boston.feature_names
df['MEDV'] = boston.target

y_data = df['MEDV']
x_data = df.drop('MEDV',axis=1)

x_data.head()

x_train, x_test, y_train, y_test = train_test_split(x_data, y_data, test_size=0.15, random_state=10)

# Model using 'RM' & 'LSTAT'
lr = LinearRegression()
lr.fit(x_train[['RM','LSTAT']],y_train)

# Prediction using Training Data
Y_trainpr = lr.predict(x_train[['RM','LSTAT']])

# Prediction using Test Data
Y_testpr = lr.predict(x_test[['RM','LSTAT']])

# Model evaluation using Test Data
mse1_test = mean_squared_error(y_test,Y_testpr)
print("Mean square error is",mse1_test)
print("R-Square value using test data is", lr.score(x_test[['RM','LSTAT']],y_test))
print("\n")

# Model evaluation using Training Data
mse1_train = mean_squared_error(y_train,Y_trainpr)
print("Mean square error is",mse1_train)
print("R-Square value using training data is", lr.score(x_train[['RM','LSTAT']],y_train))
print("\n")

Rcross = cross_val_score(lr, x_data, y_data, cv=4)

print(Rcross)
print("The mean of the folds are", Rcross.mean(), "and the standard deviation is" , Rcross.std())

Output

Mean square error using test data is 47.91753944668556

R-Square value using test data is 0.5784951912105682

Mean square error using training data is 27.636206385020866

R-Square value using training data is 0.6375993387266081

[ 0.60217169 0.60398145 0.35873597 -1.10867706]

The mean of the folds are 0.11405301290098091 and the standard deviation is 0.7129565737131626*

$\endgroup$
0
$\begingroup$

After further research, I found that the cross_val_score default behavior is that the data is not pulled randomly. Modifying the code as below gives the desired results :-

Rcross = cross_val_score(lr, x_data, y_data, cv=KFold(n_splits=4,shuffle=True))

Output :-

[0.64718237 0.71376343 0.72901413 0.72646089]

The mean of the folds are 0.704105206129402 and the standard deviation is 0.03336810520510913

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.