1
$\begingroup$

I am a 10th grade student working on a science fair project that involves making predictions about adherence given patient data. I have separated the week into 21 time slots, three for each time of day (1 is Monday morning, 2 is monday afternoon, etc.) into a day_time variable. There is also a "day" variable (1-7) and a "time" variable (1-3 (morn, aft, night). Adherence values are binary (0 means they did not take the medicine, 1 means they did). I have created a csv with 30 weeks worth of data, and have given every time slot a 1 for adherence except 3 select slots, which include the "afternoon" time slot (2 out of 1-3), the "Thursday" time slot (4 out of 1-7) and the Sunday night time slot (21 out of 1-21).These slots have all 0s except 1 or 2 exceptions. However, when I fit a Logistic Regression model to the data, the model predicts every adherence value as 1, resulting in terrible accuracy. I am using Scikit-learn and I used the class_weight = 'balanced' parameter, but this just made the accuracy worse. Yes the model began to predict more than just 1, but the accuracy was far far worse (talking below 0.5 here).

Just for fun, I simulated data in which the first 10 time slots were 1, and the remaining 11 were 0, and this pattern repeated for all 30 weeks. Logistic regression had a 100% accuracy here. This made me believe the model did so badly with my first dataset because there were many less 0s than 1s. But then, I simulated data which had roughly the same number of 0s and 1s in a week, but this time they were not all in a row. This exact pattern repeated 30 times, the model then had an accuracy of around 0.5 again. I have no idea what is causing such terrible accuracy. Is it something wrong with class weights, or the threshold? Should I not be using logistic regression? (hopefully I should be, as a right up about my model is due this Friday)

I am using scikit packages for the model.

Here is a link to the google sheets which I downloaded as a csv to use for my data: [https://docs.google.com/spreadsheets/d/1AYJQUt8LcmI3cEKwuOZB2m8EGLAKu5djYQ_Q05Y_PLo/edit?usp=sharing][1]

Here is my code:

import pandas as pd
%pylab inline
df = pd.read_csv("/Users/neelashabhattacharjee/scfair/patientdata.csv")
from sklearn.cross_validation import train_test_split
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
x = scaler.fit_transform(x)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2)

def base_rate_model(x):
     y = np.ones(x.shape[0])
     return y
y_base_rate = base_rate_model(x_test)
from sklearn.metrics import accuracy_score
print("Base rate accuracy is %2.2f" % accuracy_score(y_test, y_base_rate))

from sklearn.linear_model import LogisticRegression
model = LogisticRegression(penalty = 'l2', C = 1, class_weight = None)
model.fit(x_train, y_train)
print("Logistic accuracy is %2.2f" % accuracy_score(y_test, model.predict(x_test)))

from sklearn.metrics import roc_auc_score
from sklearn.metrics import classification_report
print("Base Model:")
base_roc_auc = roc_auc_score(y_test, base_rate_model(x_test))
print("Base Rate AUC = %2.2f" % base_roc_auc)

logit_roc_auc = roc_auc_score(y_test, model.predict(x_test))
print("Logistic AUC = %2.2f" % logit_roc_auc)

# Just some code to help me look more at the predictions of the model and compare it to the actual data

# Display because I am using jupyter notebook

predictions = model.predict(x_test)

display(predictions)

display(y_test)

# How many ones being predicted by model vs real # of ones

ones = 0
for num in y_test:
    if num == 1:
        ones = ones + 1

print("Actual # of ones: {}".format(ones))

predones = 0
for num in predictions:
    if num == 1:
        predones = predones + 1
print("Predicted # of ones: {}".format(predones))
$\endgroup$
6
  • $\begingroup$ one issue is that you are not optimising the regularisation parameter, C, suggest to use scikit-learn.org/stable/modules/generated/… $\endgroup$
    – seanv507
    Dec 30, 2018 at 0:59
  • $\begingroup$ Thanks Sean, but I have tried optimizaing the parameter and setting C = 1, but it made no difference whatsoever $\endgroup$ Dec 30, 2018 at 1:40
  • 1
    $\begingroup$ "I have tried optimizaing the parameter and setting C = 1," - optimising the parameter means trying different values of C (not setting to 1, unless optimal value is 1?) in particular for your test case I assume you need a large value of C $\endgroup$
    – seanv507
    Dec 30, 2018 at 2:09
  • 2
    $\begingroup$ you have to use dummy variables for each of your variables: scikit-learn.org/stable/modules/generated/… scikit-learn.org/stable/modules/…. in logistic regression, the 'decision' function is a linear function of the inputs, so if you want different behaviour for each possible value of day, then you need a separate input for each possible value of day... $\endgroup$
    – seanv507
    Dec 30, 2018 at 2:23
  • $\begingroup$ I turned my discrete data into continuous data using the hot encoding module like the following: from sklearn.preprocessing import OneHotEncoder enc = OneHotEncoder() enc.fit(x)x but the model is still only predicting ones $\endgroup$ Dec 30, 2018 at 18:54

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.