2
$\begingroup$

When I use a normalized values for the values of the target column in the following DL regression model I get a very good accuracy, and if I don't, the accuracy is a mess.

However I've reading that normalize the target values should not affect the behavior, so I'm confused and I don't understand what's going on in this case.

Can I please get a feedback about what I'm doing right or wrong?

I've paste the code below, nevertheless, for those of you that prefer an .ipynb view, you can visit the following: https://github.com/Mike76b/Deep-Learning-Regression-model/blob/master/BkFriday-Demo.ipynb

And if you have troubles opening the .ipynb copy the link above and paste it in the following page: https://nbviewer.jupyter.org/

Thanks.

Source of the data (kaggle) https://www.kaggle.com/mehdidag/black-friday

Modules used

import pandas as pd
import numpy as np
import keras
from keras.models import Sequential
from keras.layers import Dense
import matplotlib.pyplot as plt
%matplotlib inline

...

Using TensorFlow backend.

For explanatory purpose only:

...

bkFriday_df = pd.read_csv('D:/ProgramData/Data Resources/BlackFriday/BlackFriday.csv', sep=',')

bkFriday_df.head()
User_ID     Product_ID  Gender  Age     Occupation  City_Category   Stay_In_Current_City_Years  Marital_Status  Product_Category_1  Product_Category_2  Product_Category_3  Purchase
0   1000001     P00069042   F   0-17    10  A   2   0   3   NaN     NaN     8370
1   1000001     P00248942   F   0-17    10  A   2   0   1   6.0     14.0    15200
2   1000001     P00087842   F   0-17    10  A   2   0   12  NaN     NaN     1422
3   1000001     P00085442   F   0-17    10  A   2   0   12  14.0    NaN     1057
4   1000002     P00285442   M   55+     16  C   4+  0   8   NaN     NaN     7969

bkFriday_df.shape
(537577, 12)

bkFriday_df.columns
Index(['User_ID', 'Product_ID', 'Gender', 'Age', 'Occupation', 'City_Category',
   'Stay_In_Current_City_Years', 'Marital_Status', 'Product_Category_1',
   'Product_Category_2', 'Product_Category_3', 'Purchase'],
  dtype='object')


bkFriday_df.describe(include = 'all')
    User_ID     Product_ID  Gender  Age     Occupation  City_Category   Stay_In_Current_City_Years  Marital_Status  Product_Category_1  Product_Category_2  Product_Category_3  Purchase
count   5.375770e+05    537577  537577  537577  537577.00000    537577  537577  537577.000000   537577.000000   370591.000000   164278.000000   537577.000000
unique  NaN     3623    2   7   NaN     3   5   NaN     NaN     NaN     NaN     NaN
top     NaN     P00265242   M   26-35   NaN     B   1   NaN     NaN     NaN     NaN     NaN
freq    NaN     1858    405380  214690  NaN     226493  189192  NaN     NaN     NaN     NaN     NaN
mean    1.002992e+06    NaN     NaN     NaN     8.08271     NaN     NaN     0.408797    5.295546    9.842144    12.669840   9333.859853
std     1.714393e+03    NaN     NaN     NaN     6.52412     NaN     NaN     0.491612    3.750701    5.087259    4.124341    4981.022133
min     1.000001e+06    NaN     NaN     NaN     0.00000     NaN     NaN     0.000000    1.000000    2.000000    3.000000    185.000000
25%     1.001495e+06    NaN     NaN     NaN     2.00000     NaN     NaN     0.000000    1.000000    5.000000    9.000000    5866.000000
50%     1.003031e+06    NaN     NaN     NaN     7.00000     NaN     NaN     0.000000    5.000000    9.000000    14.000000   8062.000000
75%     1.004417e+06    NaN     NaN     NaN     14.00000    NaN     NaN     1.000000    8.000000    15.000000   16.000000   12073.000000
max     1.006040e+06    NaN     NaN     NaN     20.00000    NaN     NaN     1.000000    18.000000   18.000000   18.000000   23961.000000

...

Columns that I will use to build the model

...

Some of them
toAdd_df = pd.get_dummies(bkFriday_df[['Gender', 'Age', 'City_Category', 'Stay_In_Current_City_Years']])
toAdd_df.shape
(537577, 17)

Predictors DataFrame
All of them
pred_df = bkFriday_df[['Occupation', 'Marital_Status']].join(toAdd_df)
pred_df.shape
(537577, 19)

pred_df.head()
    Occupation  Marital_Status  Gender_F    Gender_M    Age_0-17    Age_18-25   Age_26-35   Age_36-45   Age_46-50   Age_51-55   Age_55+     City_Category_A     City_Category_B     City_Category_C     Stay_In_Current_City_Years_0    Stay_In_Current_City_Years_1    Stay_In_Current_City_Years_2    Stay_In_Current_City_Years_3    Stay_In_Current_City_Years_4+
0   10  0   1   0   1   0   0   0   0   0   0   1   0   0   0   0   1   0   0
1   10  0   1   0   1   0   0   0   0   0   0   1   0   0   0   0   1   0   0
2   10  0   1   0   1   0   0   0   0   0   0   1   0   0   0   0   1   0   0
3   10  0   1   0   1   0   0   0   0   0   0   1   0   0   0   0   1   0   0
4   16  0   0   1   0   0   0   0   0   0   1   0   0   1   0   0   0   0   1

pred_df.isnull().any()
Occupation                       False
Marital_Status                   False
Gender_F                         False
Gender_M                         False
Age_0-17                         False
Age_18-25                        False
Age_26-35                        False
Age_36-45                        False
Age_46-50                        False
Age_51-55                        False
Age_55+                          False
City_Category_A                  False
City_Category_B                  False
City_Category_C                  False
Stay_In_Current_City_Years_0     False
Stay_In_Current_City_Years_1     False
Stay_In_Current_City_Years_2     False
Stay_In_Current_City_Years_3     False
Stay_In_Current_City_Years_4+    False
dtype: bool


#Normalizing Occupation column
pred_df['Occupation'] = pred_df['Occupation']/pred_df['Occupation'].max()

...

Here is where the code of the model/questions begin

Purchase (target) column info

MIN val: 185

MAX val: 23961

MEAN val: 9333.859

STD val: 4921.022

...

#Normalizing Purchase (target) column
targ_df = bkFriday_df['Purchase']/bkFriday_df['Purchase'].max()


targ_df.head()

0    0.349318
1    0.634364
2    0.059346
3    0.044113
4    0.332582
Name: Purchase, dtype: float64

targ_df.shape
(537577,)

targ_df.isnull().any()
False


#Define classification model

def regression_model():
    # create model
    model = Sequential()
    model.add(Dense(50, activation='relu', input_shape=(19,)))
    model.add(Dense(50, activation='relu'))
    model.add(Dense(1))

    # compile model
    model.compile(optimizer='adam', loss='mean_squared_error')
    return model


# build the model
model = regression_model()


#Using a normalized target

model.fit(pred_df, targ_df, validation_split=0.3, epochs=10, verbose=2)

Train on 376303 samples, validate on 161274 samples
Epoch 1/10
 - 8s - loss: 597019.1829 - val_loss: 2.6506
Epoch 2/10
 - 8s - loss: 0.6978 - val_loss: 0.0456
Epoch 3/10
 - 8s - loss: 0.0436 - val_loss: 0.0434
Epoch 4/10
 - 8s - loss: 0.0433 - val_loss: 0.0431
Epoch 5/10
 - 7s - loss: 0.0430 - val_loss: 0.0430
Epoch 6/10
 - 8s - loss: 0.0429 - val_loss: 0.0431
Epoch 7/10
 - 8s - loss: 0.0428 - val_loss: 0.0430
Epoch 8/10
 - 8s - loss: 0.0427 - val_loss: 0.0428
Epoch 9/10
 - 8s - loss: 0.0426 - val_loss: 0.0431
Epoch 10/10
 - 8s - loss: 0.0426 - val_loss: 0.0428

<keras.callbacks.History at 0x2d64e63d4e0>


#Using a target that is NOT normalized

targ_df1 = bkFriday_df['Purchase']

model.fit(pred_df, targ_df1, validation_split=0.3, epochs=10, verbose=2)

Train on 376303 samples, validate on 161274 samples
Epoch 1/10
 - 10s - loss: 27449712.1709 - val_loss: 24691241.0429
Epoch 2/10
 - 9s - loss: 24545862.6251 - val_loss: 24694663.1769
Epoch 3/10
 - 9s - loss: 24547027.7592 - val_loss: 24691479.8413
Epoch 4/10
 - 9s - loss: 24549062.7868 - val_loss: 24693882.1764
Epoch 5/10
 - 9s - loss: 24546786.5014 - val_loss: 24694371.9963
Epoch 6/10
 - 9s - loss: 24548953.5945 - val_loss: 24693229.1358
Epoch 7/10
 - 9s - loss: 24546170.8858 - val_loss: 24692611.7169
Epoch 8/10
 - 9s - loss: 24547501.0794 - val_loss: 24700711.9410
Epoch 9/10
 - 9s - loss: 24548278.9858 - val_loss: 24690487.4462
Epoch 10/10
 - 9s - loss: 24548030.6103 - val_loss: 24702105.0564

<keras.callbacks.History at 0x2d64c66ff60>

targ_df1.dtype
dtype('int64')

#end.
Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The main problem is that you are trying to evaluating a regression problem with a classification metric (i.e. accuracy).

Accuracy tells you how many predictions you got right as a percentage of your total predictions. For example if your model makes $10$ predictions and gets $7$ of them right, you have a $70\%$ accuracy.

In regression problems, you are trying to predict a continuous value, in this case the amount of the purchase in dollars. For this you need a metric suitable for regression. These metrics show you how close you are to the target price. For example, if the target price is $9.99$ and your model predicts $10.50$ it's better than if it predicted $11.34$. If you used accuracy both of these two predictions would be counted as misses and we'd have no information over which one is better. For this reason we need a regression metric, e.g. Mean Absolute Error, Mean Squared Error, etc.

TL;DR: Don't use accuracy to evaluate regression models!

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for you answer, I thought that I was evaluating the model with the Mean Squared Error when I set loss='mean_squared_error' now I changed that line of code to model.compile(optimizer='adam', loss='mean_squared_error', metrics = ['mean_squared_error']) Is it all I had to do? and if so, could you please tell me how can I see the performance now? I mean, after running the model, what function/line of code should I use to see how well it was with the new setting. I know it might be too simple questions but is just that I'm starting to learn all these concepts and tools. Thanks rgds $\endgroup$
    – Mike
    Commented Dec 12, 2018 at 16:10
  • $\begingroup$ The performance of your model can be measured through the MSE on the validation/test set. A lower MSE means a better model (because the predictions fall closer to their actual values). $\endgroup$
    – Djib2011
    Commented Dec 12, 2018 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.