Reason for high MSE and negative R square value

Question

I am getting really high MSE and negative R square value.

Dataset: https://docs.google.com/spreadsheets/d/1moTZS_LgOn6d74NC44i9lVcWchj-abVx/edit?usp=sharing&ouid=100514649347129021200&rtpof=true&sd=true

Preprocessing and reshaping:

# All the libraries used

import numpy as np
import pandas as pd
import seaborn as sns
from sklearn import preprocessing
import matplotlib.pyplot as plt
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler,MinMaxScaler
from sklearn.linear_model import LinearRegression
from sklearn.svm import SVR
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score
from sklearn.neural_network import MLPRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split

# Reading excel sheet
dataframe = pd.read_excel('Customer-Lifetime-Value-Prediction.xlsx')


y=np.array(dataframe.iloc[:,-1])

ct=ColumnTransformer(transformers=[('encoder',OneHotEncoder(),[0,1,2,3,4,6,7,13,14,16,17])], remainder='passthrough')
data_encoded=ct.fit_transform(dataframe.iloc[:,:-1])
X=data_encoded

X=X.reshape(6817,-1)

Regression

regressor_list=[LinearRegression(),SVR(kernel = 'poly'), SVR(kernel = 'rbf'), MLPRegressor(),SVR(kernel = 'linear')]

for r in [1,20,40]:
  print('Random state',r,':')
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=r)

  scaler = StandardScaler()
  X_train_data_z = scaler.fit_transform(X_train)
  X_test_data_z = scaler.fit_transform(X_test)

  for reg in regressor_list:
    model = reg.fit(X_train_data_z, y_train)
    y_pred = model.predict( X_test_data_z)
    print("Regressor:", reg, ", method: z_score" )
    MSE = mean_squared_error(y_test,y_pred)
    R2 = r2_score(y_test,y_pred)
    print("MSE:",round(MSE,2))
    print("R2:",round(R2,2),"\n")
    print ('\n_______________________________________________________________\n')

  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=r)

  scaler = MinMaxScaler()
  X_train_data_mm = scaler.fit_transform(X_train)
  X_test_data_mm = scaler.fit_transform(X_test)

  for reg in regressor_list:
    model = reg.fit(X_train_data_mm, y_train)
    y_pred = model.predict( X_test_data_mm)
    print("Regressor:", reg, ", method: minmax" )
    MSE = mean_squared_error(y_test,y_pred)
    R2 = r2_score(y_test,y_pred)
    print("MSE:",MSE)
    print("R2:",R2,"\n")
    print ('\n_______________________________________________________________\n')

Here is the results:

Random state 1 :
Regressor: LinearRegression() , method: z_score
MSE: 3.0008567406470443e+31
R2: -6.459648482629139e+23 

_______________________________________________________________

Regressor: SVR(kernel='poly') , method: z_score
MSE: 51202109.43
R2: -0.1 

_______________________________________________________________

Regressor: SVR() , method: z_score
MSE: 51356264.15
R2: -0.11 

_______________________________________________________________

/usr/local/lib/python3.10/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:686: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  warnings.warn(
Regressor: MLPRegressor() , method: z_score
MSE: 44566013.58
R2: 0.04 

_______________________________________________________________

Regressor: SVR(kernel='linear') , method: z_score
MSE: 44154219.5
R2: 0.05 

_______________________________________________________________

Regressor: LinearRegression() , method: minmax
MSE: 40459221.498533726
R2: 0.12907422329995377 

_______________________________________________________________

Regressor: SVR(kernel='poly') , method: minmax
MSE: 50956632.87210217
R2: -0.09689320304304516 

_______________________________________________________________

Regressor: SVR() , method: minmax
MSE: 51359375.80852645
R2: -0.10556265321347125 

_______________________________________________________________

/usr/local/lib/python3.10/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:686: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  warnings.warn(
Regressor: MLPRegressor() , method: minmax
MSE: 42738481.444474116
R2: 0.08001084132678893 

_______________________________________________________________

Regressor: SVR(kernel='linear') , method: minmax
MSE: 49245824.59737017
R2: -0.06006631981915911 

_______________________________________________________________

Random state 20 :
Regressor: LinearRegression() , method: z_score
MSE: 8.545190468136494e+31
R2: -1.6600719569738018e+24 

_______________________________________________________________

Regressor: SVR(kernel='poly') , method: z_score
MSE: 56782657.69
R2: -0.1 

_______________________________________________________________

Regressor: SVR() , method: z_score
MSE: 56966340.19
R2: -0.11 

_______________________________________________________________

/usr/local/lib/python3.10/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:686: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  warnings.warn(
Regressor: MLPRegressor() , method: z_score
MSE: 47946197.05
R2: 0.07 

_______________________________________________________________

Regressor: SVR(kernel='linear') , method: z_score
MSE: 48616135.95
R2: 0.06 

_______________________________________________________________

Regressor: LinearRegression() , method: minmax
MSE: 42533920.24046921
R2: 0.17369462419061132 

_______________________________________________________________

Regressor: SVR(kernel='poly') , method: minmax
MSE: 56516489.31035551
R2: -0.097944385915965 

_______________________________________________________________

Regressor: SVR() , method: minmax
MSE: 56970395.828559555
R2: -0.10676241618419491 

_______________________________________________________________

/usr/local/lib/python3.10/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:686: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  warnings.warn(
Regressor: MLPRegressor() , method: minmax
MSE: 47189864.93344831
R2: 0.08324370625195998 

_______________________________________________________________

Regressor: SVR(kernel='linear') , method: minmax
MSE: 54503505.91413023
R2: -0.05883820919131444 

_______________________________________________________________

Random state 40 :
Regressor: LinearRegression() , method: z_score
MSE: 9.203880539100096e+30
R2: -1.8587470874246684e+23 

_______________________________________________________________

Regressor: SVR(kernel='poly') , method: z_score
MSE: 55388763.65
R2: -0.12 

_______________________________________________________________

Regressor: SVR() , method: z_score
MSE: 55564439.09
R2: -0.12 

_______________________________________________________________

/usr/local/lib/python3.10/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:686: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  warnings.warn(
Regressor: MLPRegressor() , method: z_score
MSE: 49991480.86
R2: -0.01 

_______________________________________________________________

Regressor: SVR(kernel='linear') , method: z_score
MSE: 48057332.09
R2: 0.03 

_______________________________________________________________

Regressor: LinearRegression() , method: minmax
MSE: 42741071.18572825
R2: 0.13683319505958647 


_______________________________________________________________

Regressor: SVR(kernel='poly') , method: minmax
MSE: 55060433.90474105
R2: -0.1119594688131671 


_______________________________________________________________

Regressor: SVR() , method: minmax
MSE: 55542005.49753461
R2: -0.12168492963036082 


_______________________________________________________________

/usr/local/lib/python3.10/dist-packages/sklearn/neural_network/_multilayer_perceptron.py:686: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
  warnings.warn(
Regressor: MLPRegressor() , method: minmax
MSE: 45639416.499979034
R2: 0.07830037416596236 


_______________________________________________________________

Regressor: SVR(kernel='linear') , method: minmax
MSE: 53392285.66973744
R2: -0.07827079086875832

How can I get low MSE and good R square value.

So the total number of experiments is: 2(normalizations) x 3(data splits) x 5(regressors) = 30 experiments

I used all of the following regression techniques: a. Linear regression b. SVM with linear kernel c. SVM with polynomial kernel d. SVM with RBF kernel e. Neural Networks

None of the MSEs are low. You may get more clarity if you start with one model, say regression, and learn how to diagnose issues with / improve that model in incremental steps. — dipetkov, Commented May 13 at 19:36
You need to apply to the test data the preprocessor fitted on the training data, i.e., simply use the transform() method on the test data, instead of refitting it to the test data. This is a conceptual mistake that can easily be fixed. A second problem of the linear model is that representing features in a linear way is usually overly naive. One needs spline transforms, interactions, dummy encodings etc. to make a strong model. — Michael M, Commented May 13 at 19:48

Michael M · Accepted Answer · 2024-05-13 19:53:28Z

1

You need to apply to the test data the preprocessor fitted on the training data, i.e., simply use the transform() method on the test data, instead of refitting it to the test data. This is a conceptual mistake that can easily be fixed.
A second problem of your linear model is that representing features linearly is usually overly naive. One needs spline transforms, interactions, dummy encodings etc. to make a strong model.

answered May 13 at 19:53

Michael M

12.1k5 gold badges35 silver badges50 bronze badges

Add a comment |

Stack Exchange Network

Reason for high MSE and negative R square value

Regression

1 Answer 1

Your Answer

Linked

Hot Network Questions

Reason for high MSE and negative R square value

Regression

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Linked

Related

Hot Network Questions