Which type of labels should I use/gather for logistic regression?

Question

I currently have a dataset of drawings, each drawing being represented by some features. Each feature (independent variable) is a continuous number. None of the drawings have a label as of yet, which is why I am planning to start a sort of questionaire with people. However, before I can correctly setup such questionaire, I should have an idea of what kind of labels I should use for my training data.

At first thought, I was thinking about letting people rate the drawings on a scale, for example from 1 to 5 with 1 being bad, 3 being average and 5 being good. Alternatively, I could also reduce the question to a simple good or bad question. The latter would mean I lose some valuable information, but the dependent variable could then be considered 'binary'.

Using the training data I then composed, I would need to have a machine learning algorithm (model) which given a drawing, predicts if the drawing is good or not. Ideally, I would have some way of tuning the strictness in this prediction. For example, the model could instead of simply predicting 'good' or 'bad', predict the likelyhood of a painting being good on a scale of 0 to 1. I could then say "Well, let's say all paintings which are 70% likely to be good, are considered as good". Another example would be that the model predicts the goodness using the same categorical values the people used to rate the drawing initially. So it would either predict the drawing being a 1, 2, 3, 4 or 5. Similar to my first example, I could then say "Well, all paintings which are rated at least a 4, are considered good paintings" and tune this threshhold to my liking.

After doing some research, I came up with logistic and linear regression being good candidates. However, if which of the two would be the best for my scenario? Equally important, how would I need to format my labels? Just simple 0's and 1's or a scale?

Answer 1

The “awful-bad-average-good-excellent” rating is totally fine. It’s called an original variable, and there are many models for such a response variable. The simplest one would be an ordinal logistic regression, which is kind of a hybrid of regression and predicting category probability (classification).

In a multiclass (multinomial) logistic regression, say with the MNIST digit images, there is no difference in penalty for predicting a high probability of a $1$ digit as a $5$ When the digit is a $2$, but if $1-5$ are your awful-bad-average-good-excellent categories, it’s worse to call your $2$ (bad) a $5$ (excellent) than a $1$ (awful). Treating the response variables as an ordinal variable rather than purely categorical allows for the parameter optimization to consider this.

Answer 2

Welcome! If your aim is to classify into good or bad, having a binary variable is the best option. On the other hand if you want to have more classification within good (good to best) or bad(neutral to worst), you can have 5 point scale.

With your idea of having the ratings 4 and above as good and the rest as bad is not justifying the efforts put forth by the respondents for filling the questions (they need to think a fraction of a second more than that they do for binary response). Moreover, the respondents would be feeling easier to answer good or bad than to answer a five point scale. Unless, you want to draw more inference from the responses, you can go for binary.

If you are going for binary response, binary logistic regression would be a better option.

Answer 3

It is up to you to choose whether to predict a binary 'good' vs 'bad or to predict a score such as on a scale of 1 to 5. If you choose binary classification, logistic regression would be your answer.

If you do choose to represent the score as on a scale of 1 to 5, I would not represent these numbers as categories in a flat categorical classification problem. When minimizing a classification error such as categorical cross entropy, the resulting model has no sense of scale, in that the score '1' is no closer to '2' than it is to '5'. I would train a regression model and have it predict a continuous number, even though the training data is rounded to whole numbers.