6
$\begingroup$

I am using SPSS to analyze a data set which aims to predict whether individuals have cancer based on five symptoms (a, b, c, d, e). In this data set most individuals have cancer. I ran a Binary Logistic Regression and got the following output:

Block 0

This tests the model with which only includes the constant, and overall it predicted 91.8% correct. I understand that the fact that I have significant predictors in the "Variables not in the Equation" table means that the addition of one or more of these variables to the model should improve its predictive power.

I then looked at the model after all the predictors were included:

http://i.imgur.com/Y5WdePl.png

The prediction is only very slightly different. Now it's predicting that two individuals will not have cancer. The overall percentage correct remains 91.8%.

  • Why did no improvement occur, despite the predictors being significant?
  • Where should I go from here with this data set? Is it possible to improve the predictive power of the model without including new predictors?
  • How should I assess the model? Is the fact that it doesn't improve over the model with only the constant evidence that the model is useless?

Full output viewable here.

Data set downloadable here as a google doc.

Improve this question
$\endgroup$
8
  • 3
    $\begingroup$ For a start use a proper scoring rule to assess model performance, unless you can justify that 0.5 cut-off. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Apr 16, 2014 at 17:32
  • $\begingroup$ Under what circumstances might a cut-off of 0.5 be justified? I used that just because it was the SPSS default, although I do not understand why it was the default. $\endgroup$
    – user1205901 - Слава Україні
    Apr 16, 2014 at 21:44
  • 2
    $\begingroup$ There is a lot that needs to be said here. but to keep simple: (1) the sample size issue. You have 27 events...this is going to make it challenging creating any model, and I would not pursue a multivariate model with so few events. (2) as noted above and by Harrell using accuracy is inappropriate. In the biomedical field this is accepted, so I'm surprised to see it with this data (3) the comments below regarding validation -especially split sample - are wrong and I'd google to find one of Steyerberg's articles on validation to get an appropriate explanation. $\endgroup$
    – charles
    Apr 17, 2014 at 1:57
  • 1
    $\begingroup$ @user1205901: It's silly unless the sole purpose of the model is to decide between alternative courses of action & the cost of making the wrong decision is the same for each. (If the alternatives were to carry out a blood test or to do nothing; & a patient's symptoms indicated a 49% chance of his having cancer, would you really send him home?) Even then, a model that predicts a 55% chance of cancer in someone who's in fact healthy is unlucky; a model that predicts a 99.99% chance is suspect. Perhaps the main reason to use classification accuracy is to allow comparison of logistic ... $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Apr 17, 2014 at 11:34
  • $\begingroup$ ... regression models with other classification techniques that don't provide a predicted probability. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Apr 17, 2014 at 11:35

3 Answers 3

Reset to default
11
$\begingroup$

Summary

You appear to be looking at the associations between symptoms (a, b, c, d, and e, coded as linear, numeric variables) and cancer status (yes versus no, coded in binary).

Associations versus predictions

I think you are looking at associations between the symptoms and cancer status rather than the ability of the symptoms to predict cancer status. If you wanted to really investigate predictive ability, you would need to divide your data set in half, fit models to one half of the data, and then use them to predict the cancer status of the patients in the other half of the data set. Note that this describes the simplest case of validation of a model using a single data set. You shouldn't actually do this. What you could really do is employ n-fold cross validation (for example, using the rms package in R) to make the most efficient use of your data.

Starting off

You may have already done this, but prior to playing around with logistic regression modeling I think you should take a step back and just look at your data. Using the program R to compute a few basic summary statistics...

# Load libraries
library(Rmisc)
library(metafor)

# Load data
data <- read.csv("example_data.csv", header = TRUE, na.strings = "")
attach(data)

# Summarize data
summary(data)
       a              b               c               d               e             cancer      
 Min.   :11.0   Min.   :13.00   Min.   :13.00   Min.   :12.00   Min.   :17.00   Min.   :0.0000  
 1st Qu.:19.0   1st Qu.:27.00   1st Qu.:28.00   1st Qu.:36.00   1st Qu.:33.00   1st Qu.:1.0000  
 Median :24.0   Median :31.00   Median :32.00   Median :40.00   Median :38.00   Median :1.0000  
 Mean   :24.8   Mean   :31.39   Mean   :32.44   Mean   :39.39   Mean   :37.71   Mean   :0.9169  
 3rd Qu.:30.0   3rd Qu.:36.00   3rd Qu.:37.00   3rd Qu.:43.50   3rd Qu.:42.00   3rd Qu.:1.0000  
 Max.   :49.0   Max.   :50.00   Max.   :50.00   Max.   :50.00   Max.   :50.00   Max.   :1.0000  
 NA's   :20     NA's   :18      NA's   :21      NA's   :20      NA's   :20      NA's   :6

And now to plot some exploratory scatter plots... Pay attention to any linear relationships between variables that pop out to your eye. Also pay attention (as Benjamin mentioned below) to the plots of the symptom variables versus cancer status.

plot(data)

Scatter plots

And look at some histograms to get a sense of the distribution of your data... Always good to do this before plugging them into a regression model

 hist(data)

Histograms

Going a bit further...

I would compute the mean and 95%CI for each symptom variable and stratify them by cancer status and plot those... Just by looking at this you will know visually which variables are going to be significant in your logistic regression model. Here I just plot the data...

forest(
x = c(24.44636,28.94667,31.63066,28.62963,32.59910,30.65852,39.79738,35.04111,37.99030,34.41185),
ci.lb = c(23.57979,25.72939,30.84611,26.15883,31.88579,28.52778,39.16493,32.27390,37.26171,32.10734),
ci.ub = c(25.31292,32.16395,32.41520,31.10043,33.31242,32.78926,40.42983,37.80832,38.71888,36.71637),
xlab = "Mean and 95% CI", slab = c("a cancer","a healthy","b cancer","b healthy","c cancer","c healthy","d cancer","d healthy","e cancer","e healthy"))

Forest plot

Looking at the plot above, you get a visual sense of the fact that you have way more cancer patients contributing to the data set than non-cancer patients.

Last...

I would just compute univariate effects estimates for each symptom variable for their associations with cancer outcome. Then I would multiply all of the resultant p values by five, since you are doing that many exploratory tests. You can do that in SPSS easily. For the results of the models, I would focus more on the direction, magnitude, and confidence intervals for the resultant effects estimates. Below I have plotted the effects estimates and their confidence intervals from univariate models of each separate symptom variable... Now you should go build models that are adjusted for age, gender, smoking, etc. and make another plot like this... I do agree with Benjamin that there is probably not a whole lot you can likely learn from these data given the paucity of healthy controls.

Logistic regression results

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Is it reasonable (in principle) for me to model all the data using logistic regression, if I'm planning to later collect another data set and attempt to use the model to predict the cancer status of people in that data set? $\endgroup$
    – user1205901 - Слава Україні
    Apr 16, 2014 at 21:50
  • $\begingroup$ I would calculate the univariate risk estimate, 95% confidence interval, and p value for each symptom variable separately using logistic regression, and then plot them all on a forest plot so you can visualize them graphically. Then I would produce another set of five models adjusted for known clinical predictors for whatever cancer you are studying. If you were studying lung cancer, for example, I would include patient age, gender, clinical stage, and smoking status. Once you have adjusted for these known associations, I would look at the effects estimates for the symptom variables again. $\endgroup$
    – Alexander
    Apr 16, 2014 at 22:46
  • $\begingroup$ If they do end up to be independently associated with the outcome, perhaps you could contemplate building a model that contained all of the significant symptom variables (in addition to the known clinical predictors of course). You would have to validate it on an outside data set, however. Out of curiosity, what symptoms do these variables refer to? I can't think of many symptoms that can be transformed to a numeric scale like this. $\endgroup$
    – Alexander
    Apr 16, 2014 at 22:49
  • $\begingroup$ Also, you definitely should not use a step-wise regression model building algorithm. They tend to find arbitrary associations in the data (by plucking and removing different combinations of predictor variables) and produce models that don't validate well in replication studies. They remove the act of critical thinking from the model building process. $\endgroup$
    – Alexander
    Apr 16, 2014 at 22:52
5
$\begingroup$

One thing to check is whether there is a linear relationship between the log odds of cancer and each of your 5 predictor variables. This is an assumption in logistic regression. If this does not hold you might want to consider adding higher order terms to the model, or even a nonlinear relationship between log odds of cancer and some of the variables (by fitting a generalized additive model).

From your output, it looks like these 5 predictors do not do a good job of classifying cancer vs non-cancer.

I'll take a look at the data and add more to this question later.

After taking a look at the data I have confirmed that indeed these variables are terrible at predicting cancer. If you plot the variables against cancer status you will see that, although for some of them the non-cancer patients have a little less variability, there is very little difference between the cancer and non-cancer patients. For example:

enter image description here

So if you told me that you had a patient who had a C variable of 30...I would have no idea if that is a cancer patient or a non-cancer patient.

A bit more about your output: When you don't add any variables in it says you correctly predict 91.8% of the patients. The next table that lists significance values for adding in more variables means if you add them in one at a time.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Are you sure that just using variable D gets 99.39% accuracy? I just reran the logistic regression and as far as I can see from i.imgur.com/rD8aPlJ.png the accuracy went to 91.5% (that is, it got slightly worse). $\endgroup$
    – user1205901 - Слава Україні
    Apr 16, 2014 at 21:40
  • $\begingroup$ Ah you are right, I just made an arithmetic error. But anyway you cut it, these variables don't discriminate your data very well. Since most people have cancer in this data set you can do just as well at predicting whether they have cancer by just saying they all have it. $\endgroup$
    – bdeonovic
    Apr 16, 2014 at 21:53
4
$\begingroup$

Ignore the classification tables completely. They are not based on sound statistical methods, and are completely arbitrary.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I would be keen to hear from @frank-harrell, or anyone else, on the separate but related question of whether classification tables are based on bound statistical methods. I've been working on the assumption that they're worth paying attention to. $\endgroup$
    – user1205901 - Слава Україні
    Apr 16, 2014 at 21:27
  • 2
    $\begingroup$ This was never a sound idea. I very much regret including a classification table as an example in the users guide for the first SAS procedure for logistic regression. In my view classification tables represent bad statistical practice. $\endgroup$
    – Frank Harrell
    Apr 16, 2014 at 22:16
  • $\begingroup$ Could you explain a little more about why they are bad? $\endgroup$
    – user1205901 - Слава Україні
    Apr 16, 2014 at 22:18
  • 3
    $\begingroup$ Any method that dichotomizes a continuous variable (in this case the logistic model's predicted probability) is highly problematic. Now couple that with the fact that the threshold is completely arbitrary and the proportion classified correctly is an improper scoring rule (i.e., it is optimized by a bogus model) you have a perfect storm. $\endgroup$
    – Frank Harrell
    Apr 16, 2014 at 22:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.