I am a 10th grade student working on a science fair project that involves making predictions about adherence given patient data. I have separated the week into 21 time slots, three for each time of day (1 is Monday morning, 2 is monday afternoon, etc.) into a day_time variable. There is also a "day" variable (1-7) and a "time" variable (1-3 (morn, aft, night). Adherence values are binary (0 means they did not take the medicine, 1 means they did). I have created a csv with 30 weeks worth of data, and have given every time slot a 1 for adherence except 3 select slots, which include the "afternoon" time slot (2 out of 1-3), the "Thursday" time slot (4 out of 1-7) and the Sunday night time slot (21 out of 1-21).These slots have all 0s except 1 or 2 exceptions. However, when I fit a Logistic Regression model to the data, the model predicts every adherence value as 1, resulting in terrible accuracy. I am using Scikit-learn and I used the class_weight = 'balanced' parameter, but this just made the accuracy worse. Yes the model began to predict more than just 1, but the accuracy was far far worse (talking below 0.5 here).
Just for fun, I simulated data in which the first 10 time slots were 1, and the remaining 11 were 0, and this pattern repeated for all 30 weeks. Logistic regression had a 100% accuracy here. This made me believe the model did so badly with my first dataset because there were many less 0s than 1s. But then, I simulated data which had roughly the same number of 0s and 1s in a week, but this time they were not all in a row. This exact pattern repeated 30 times, the model then had an accuracy of around 0.5 again. I have no idea what is causing such terrible accuracy. Is it something wrong with class weights, or the threshold? Should I not be using logistic regression? (hopefully I should be, as a right up about my model is due this Friday)
I am using scikit packages for the model.
Here is a link to the google sheets which I downloaded as a csv to use for my data: [https://docs.google.com/spreadsheets/d/1AYJQUt8LcmI3cEKwuOZB2m8EGLAKu5djYQ_Q05Y_PLo/edit?usp=sharing][1]
Here is my code:
import pandas as pd
%pylab inline
df = pd.read_csv("/Users/neelashabhattacharjee/scfair/patientdata.csv")
from sklearn.cross_validation import train_test_split
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
x = scaler.fit_transform(x)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2)
def base_rate_model(x):
y = np.ones(x.shape[0])
return y
y_base_rate = base_rate_model(x_test)
from sklearn.metrics import accuracy_score
print("Base rate accuracy is %2.2f" % accuracy_score(y_test, y_base_rate))
from sklearn.linear_model import LogisticRegression
model = LogisticRegression(penalty = 'l2', C = 1, class_weight = None)
model.fit(x_train, y_train)
print("Logistic accuracy is %2.2f" % accuracy_score(y_test, model.predict(x_test)))
from sklearn.metrics import roc_auc_score
from sklearn.metrics import classification_report
print("Base Model:")
base_roc_auc = roc_auc_score(y_test, base_rate_model(x_test))
print("Base Rate AUC = %2.2f" % base_roc_auc)
logit_roc_auc = roc_auc_score(y_test, model.predict(x_test))
print("Logistic AUC = %2.2f" % logit_roc_auc)
# Just some code to help me look more at the predictions of the model and compare it to the actual data
# Display because I am using jupyter notebook
predictions = model.predict(x_test)
display(predictions)
display(y_test)
# How many ones being predicted by model vs real # of ones
ones = 0
for num in y_test:
if num == 1:
ones = ones + 1
print("Actual # of ones: {}".format(ones))
predones = 0
for num in predictions:
if num == 1:
predones = predones + 1
print("Predicted # of ones: {}".format(predones))
