# How can I improve this basic Classification model? Have I implemented it correctly and validated the data?

I'm a student that is new to this field, I've played with the GUI version of Weka and made Neural Nets in that with premade datasets but now is the first time I've implemented one using Keras (Theano Backend) in Python. What I'm trying to do is create a model that will find a correlation between Tweet sentiment and Stock price of a single company. In this case study, I decided to use Tesla.

Here is a sample of what data I'm collecting in my DB.

Now I only using the Tweet sentiment and the stock price for my NN, What I'm using the NN for is Multi-Classification to tell me whether the stock is going up or down or staying the same. here is my code

from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import SGD
import pymysql as mysql
import pandas as pd
import config

##This is getting the data from MySQL
try:
sql = "SELECT stock_price, sentiment from tweets WHERE stock_price != 301.44"
con = mysql.connect(config.dbhost, config.dbuser, config.dbpassword, config.dbname, charset='utf8mb4', autocommit=True)
finally:
con.close()

##This is finding the % change between the stock prices. a negative number mean it has drops and positive number mean it has rissen
results['pct_chg'] = results['stock_price'].pct_change()
results['pct_chg'][0] = 0

##I then make my own One Hot Encoding in the loop below.
for index, row in results.iterrows():
if row['pct_chg'] > 0:
row['pct_chg'] = 1
if row['pct_chg'] < 0:
row['pct_chg'] = -1
if row['pct_chg'] == 0:
row['pct_chg'] = 0

#This is the ANN Model
model = Sequential()
##2 Layers to predict if the stock is going up or down or staying the same

##I delete the pct_chg because I already have it in the One Hot Encoded, I make it into its own predictor or 'pred' var
pred = results['pct_chg']
del(results['pct_chg'])
del(results['stock_price'])

sgd = SGD(lr=0.05, momentum=0.3, decay=0.08, nesterov=True)

model.compile(loss='sparse_categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

model.fit(results.as_matrix(), pred.as_matrix(), validation_split=0.1, shuffle=False)


When I get the data from using "SELECT * FROM tweets" becuase of the market is only open a certain time period the majority of the data stocks don't move so I get a 0.0 percent change, so for this post I have included the "WHERE stock_price 1= 301.44"

When I get the data from the DB the data looks like this:

100       299.58   0.045455
101       299.62   0.000000
102       299.63   0.043478
103       299.60   0.000000
104       299.60   0.000000
105       299.60   0.000000
106       299.60   0.000000
107       299.51   0.000000
108       299.51  -0.071429
109       299.51   0.045455
110       299.60   0.038462
111       299.70   0.000000
112       299.82  -0.200000
113       299.82   0.000000
114       299.82   0.000000
115       299.82   0.000000
116       299.82   0.000000
117       299.82   0.000000
118       299.84   0.000000
119       299.88   0.038462


Then after I perform pct_chg() method data it looks like this:

    stock_price  sentiment   pct_chg
100       299.58   0.045455  0.000000
101       299.62   0.000000  0.000134
102       299.63   0.043478  0.000033
103       299.60   0.000000 -0.000100
104       299.60   0.000000  0.000000
105       299.60   0.000000  0.000000
106       299.60   0.000000  0.000000
107       299.51   0.000000 -0.000300
108       299.51  -0.071429  0.000000
109       299.51   0.045455  0.000000
110       299.60   0.038462  0.000300
111       299.70   0.000000  0.000334
112       299.82  -0.200000  0.000400
113       299.82   0.000000  0.000000
114       299.82   0.000000  0.000000
115       299.82   0.000000  0.000000
116       299.82   0.000000  0.000000
117       299.82   0.000000  0.000000
118       299.84   0.000000  0.000067
119       299.88   0.038462  0.000133


After my One-Hot encoding loop, the data looks like this

     stock_price  sentiment  pct_chg
100       299.58   0.045455      0.0
101       299.62   0.000000      1.0
102       299.63   0.043478      1.0
103       299.60   0.000000     -1.0
104       299.60   0.000000      0.0
105       299.60   0.000000      0.0
106       299.60   0.000000      0.0
107       299.51   0.000000     -1.0
108       299.51  -0.071429      0.0
109       299.51   0.045455      0.0
110       299.60   0.038462      1.0
111       299.70   0.000000      1.0
112       299.82  -0.200000      1.0
113       299.82   0.000000      0.0
114       299.82   0.000000      0.0
115       299.82   0.000000      0.0
116       299.82   0.000000      0.0
117       299.82   0.000000      0.0
118       299.84   0.000000      1.0
119       299.88   0.038462      1.0


I then have my predictions array (pred) and then my results which I delete everything apart from the sentiment as I have already got what I needed from the stock_price which is the one hot encoding tell whether it is going up or down.

I then fit pred & results into the model via the numpy.as_matrix() method.

I have added a few Dropout layers as my model is very over fitted I kept getting 95% accuracy this is very unlikely, I told my lecture and she said this.

"That is ok, but again, you need to validate your machine learning models using k-fold cross-validation and testing it using a separate validation data set."

I have used the validation_split param in the model.fit() method is this what she meant? also what is K-Fold CV? I know what Cross Validation is but not the K-Fold part?

Also how would I go about making a validation set?

Lastly, How could I improve my model? What would you suggest?

Thanks.

• Here you are: k-fold CV – darXider Apr 27 '17 at 17:01