3
$\begingroup$

I am performing ordinal regression (6 response categories) on a large dataset (>15000 points) and comparing two models, one with one predictor X1 (Model 1) and the other with two predictors X1 and X2 (Model 2). The models I fit are also cumulative over the categories: as in, I am looking at outcomes being at least at a certain level - and the probability of being at level 1 is always 1.

The likelihood ratio test reveals that Model 2 provides a significantly better fit to the data when compared to Model 1.

However, I am also computing the models' respective accuracies, by summing up the number of correct estimations for each category (again cumulative) over the total number of outcomes, and find that accuracies are lower for Model2 than for Model 1.

X2 has got missing values, so I tried to run the regression again on a subset of the data which had values for all X1 and X2, when I do this the accuracy remains the same.

I am very puzzled by these results. How can the "better" model have a lower accuracy? I checked my code a hundred times for programming issues without success, so now I am starting to wonder whether there might be another possible reason for this.

Does missing data in a predictor affect the accuracy - without affecting the deviance? Are there any issues with large datasets when it comes to computing these measures?

$\endgroup$

1 Answer 1

1
$\begingroup$

The deviance is measuring the errors on the ordinal scale. If the truth is a $2$ and you predict $3$, that does not incur as much of a penalty as would a prediction of $4$.

However, when you look at accuracy, both of those are equally wrong.

I would guess that the model with higher accuracy but also higher deviance has more large misses.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.