I've been having an argument with a friend of mine, and it's very possible I'm wrong here.
We are performing binary logistic regression on a dataset with 10000 observations, classifying action as "good" or bad". There are two independent variables (x1, x2), and class variable (y, with values "good" or "bad"). In this dataset, we have 7,500 observations classified as "bad" and 2,500 classified as "good". This is because there are several different ways for a user to perform a "bad" action, but only one way for them to perform a "good" action.
We are doing our analysis in R using the glm()
function.
We create training data by randomly sampling 7,500 observations from the dataset, and create test data from the other 2,500 observations. we then build a model using binary logistic regression on the training data, then test it on the test data. The accuracy of our model is 75%.
Can we say our model is better than guessing?
He says that this model is no better than guessing. Even though the error rate is better than 50%, because the original data had a prevalence of "bad" classifiers, we would need our model to predict better than 75% in order to say it performs better than random guessing.
I disagree...but I can't defend my point with anything other than "that doesn't seem right". Can someone shed some light on the correct interpretation, and the reason for it?