Suppose I have a variable X1, which is a score on some psychological test, that indicates whether a person will commit a crime. The variable Y is a binary outcome which indicates whether a person finally committed a crime after the test.

If I create a table by using X1 and Y, the result is shown as below.

        TEST SCORE X1                       
        0   1   2   3   4   5
Y   0   100 20  10  7   8   9
    1   6   10  4   3   2   1

The summary of logistic regression and linear regression shows X1 is statistically significant.

My question is that I wonder which of these options is the appropriate answer for this result.

  1. X1 successfully indicates whether the person will commit a crime

  2. With appropriate weight (slope coefficient) X1 will successfully indicate whether the person will commit a crime.

  3. Relying on the test score is insufficient since the error rates are approximately 45% (20 + 10 + 7 + 8 + 9 + 6 + 4 + 3 + 2 + 1) / sum of all. However, appropriate weight (slope coefficient) X1 will successfully indicate whether the person will commit a crime.

Any comments or answers will be much appreciated.

  • 5
    $\begingroup$ There is no correct interpretation of the results that contains "successfully indicate the person will commit a crime or not" unless it's preceded with something like "cannot". An increase in estimated probability with score doesn't alter the fact that (1) the probability is still small for every test score, and (2) the model isn't causal in the way the wording seems to suggest. $\endgroup$ – Glen_b Mar 20 '15 at 9:50
  • 1
    $\begingroup$ I see no connection to "anova" beyond an implicit link through regression. I've removed that tag, which I think doesn't help people who might answer this question or people interested in anova. $\endgroup$ – Nick Cox Mar 20 '15 at 10:04
  • 1
    $\begingroup$ @Glen_b I think your comment is really close to an answer. Do you want to add a little bit to it and make it an answer, or should I? $\endgroup$ – Peter Flom Mar 20 '15 at 10:08
  • 3
    $\begingroup$ @Peter If you see a way you can use that comment in an answer, you should by all means use it. I didn't quite feel I was answering the question there and was pondering on what else could be said, but hadn't come to any good conclusion. I think comments that partly address the question are free game for anyone wanting to write an answer. $\endgroup$ – Glen_b Mar 20 '15 at 10:11

(NOTE: corrected; some details in original posting were based on mistyping the data into my software.)

This may be self-study; if you so should declare it. But imagining that this were real, or realistic, what you have done appears to be to treat the test score as if it were a measurement (which would need to be discussed and defended) and then use logit (logistic) and linear regression predicting criminal or not from test score. Beyond agreeing that the results are (just!) significant at conventional levels, an important piece of evidence is a graph like this:

enter image description here

The red line is a linear regression result (so-called linear probability model) and the blue curve is a logit regression result. The two are close, and there is a secondary issue of which should be used, on which I almost always plump for logit. What would happen if scores were higher is purely speculative, as you have not indicated whether higher scores than 5 are possible and in any case there are no data.

But on the major question of appropriate wording, I agree with @Glen_b: any claim of "success" would be far too strong here. Again, taking the context of detecting crime literally, that context is one in which either wrong decision (convicting the innocent or letting the guilty go free) would be a very serious matter. You would need massive and repeated analyses showing that certain test scores implied very high probability of crime for this even to be worth discussing in research literature, let alone used for real. Even for high test scores the predicted probability is not even 0.3. (Looking at the details of success or not in individual cases is also important, as you are aware; your own particular success measure seems puzzling, but I won't expand on that.)

I suspect that the example is just invented, but whether intentionally or not it does underline that there can be a massive gap between statistical significance and even borderline practical plausibility.

Note. I have tinkered with your wording and hope to have respected your meaning. However, using the slope coefficient as a weight is odd wording, as there can be no other way to use regression of any flavour!

A good comment from @kjetil prompts a further look at the data. On the graph are also empirical probabilities for each distinct score as diamonds. The significant fit of both models is indeed not even matched by a consistent monotonic trend. The very different numbers at each score level have also, I think, a side-effect of making the fits appear better than they really are. So, the more you look, the less there is a trustworthy pattern here.

  • 2
    $\begingroup$ Can those curves be correct? there is from curve close to 0.5 probability for $x=5$, but from OP's table, for $x=5$ there is 9 0's and just one 1? $\endgroup$ – kjetil b halvorsen Mar 20 '15 at 11:18
  • 1
    $\begingroup$ @kjetil b halvorsen Good question! I will expand my answer. $\endgroup$ – Nick Cox Mar 20 '15 at 12:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.