# How to reduce the number of labels in regression

I'm working in a regression problem, related to bio-signals, where my labels are integer numbers between 0 and 10. I've tried a couple of regression algorithms already, mainly linear regression.

Edit: More precisely my dataset is composed by 105 rows, and I have 40 features (real numbers). The output Y has 10 ordered values (0, 1, 2, ... 10) which represent the level of anxiety (it's the answer of a questionnaire).

The output are in on a interval scale and are not just ordinal.

Due to the nature of the problem, me and my collegues have considered the possibility of reducing the output, from a range of 10 values to 3 different output. Something like 1, 2, 3 or for example "low", "medium", "high".

Edit: Basically I want to reduce the size of the interval of the output.

My questions are:

• Once the discretization (reduction of number of labels) has been performed, should we still use regression algorithms, or since the low number of output we are in a classification scenario?
• Which is the best way to reduce the number of labels?
• Would it have sense to continue using regression and then categorize the prediction output in three different levels?
• You are using very nonstandard terminology for regression. Please state your sample size, the number of distinct values of Y, the number of candidate predictors, and the types of predictors you have. State whether the numeric variables can be considered to be on interval scales or can only be assumed to be ordinal. Jul 24, 2016 at 19:55
• I've edited the question. Please let me know if it's clear now. Jul 24, 2016 at 20:08
• The dependent variable in regression is a numerical value (a real number). Therefore, a simple approach to your problem could be discretizating after doing the regression analysis. Of course, this only would make sense if approximating your 10 values scale by a continuous variable maked sense.
– Pere
Jul 24, 2016 at 20:37
• 40 features and 105 rows? To be honest, I don't think you are going to get good results no matter what you do unless there is a very simple relationship (e.g. linear) between the features and the output. Jul 24, 2016 at 22:08
• I know that ... unfortunately when there's the need to deal with bio-signals, it's really hard to find patients who are going to participate in measurements. ... let's hope to find something. Jul 25, 2016 at 20:33

Never reduce the number of levels of an ordered or continuous variable. This results in loss of power and precision and increase in instability. For your problem an ordinal regression model such as the proportional odds model is called for. To handle the fact that you have too many candidate predictors for your effective sample size, use one of the following approaches:

1. penalized maximum likelihood estimation to effectively reduce the number of parameters being estimated
2. data reduction (unsupervised learning) that is masked to Y; here you collapse the predictor space using things like variable clustering, principal components, or redundancy analysis. This usually helps in interpretation also.

See my course notes for more details.

Note that your output variable Y is probably not interval scaled.

2. This depends on the problem that you are solving. You should figure out the most reasonable way to define "low", "medium", and "high" values such that it makes sense in the underlying theory of the problem (bio-signals in your case). You can find two critical values, $\alpha$ and $\beta$, such that the output is considered low if it is less than $\alpha$, normal if it is between $\alpha$ and $\beta$, and high if it is greater then $\beta$. You can do this by constructing distribution of outputs on the current scale ($1,2,\ldots,10$), and then determine $\alpha$ and $\beta$ as percentiles (for example, let $\alpha$ be the 10th percentile and $\beta$ the 90th). It is hard to give any solid advice on actual values of $\alpha$ and $\beta$ without knowing the nature of data. Maybe you can even use fuzzy set theory to define membership functions of "low", "medium" and "high", and then use them to determine output label of a concrete example. There are several alternatives. :)