# Is there any algorithm combining classification and regression?

I'm wondering if there's any algorithm could do classification and regression at the same time. For example, I'd like to let the algorithm learn a classifier, and at the same time within each label, it also learns a continuous target. Thus, for each training example, it has a categorical label and a continuous value.

I could train a classifier first, and then train a regressor within each label, but I'm just thinking that if there's an algorithm that could do both, it would be wonderful.

• Have a look at Unbiased Recursive Partitioning Nov 14, 2016 at 19:05
• @Tim, [finite-fixture-model] (a tag that you used here) does not have an excerpt. Care to provide one? Nov 15, 2016 at 12:39
• Shudong, did you mean "classification" or did you mean "clustering"? You asked about classification but then accepted Tim's answer which is about clustering. Nov 15, 2016 at 12:40
• @amoeba done, it's not perfect byt maybe someone, someday will improve the excerpt.
– Tim
Nov 15, 2016 at 13:03
• @amoeba I mean classification. Tims answer is something close. There's no other answers. Maybe I should wait a bit? Nov 15, 2016 at 13:48

The problem that you are describing can be solved by latent class regression, or cluster-wise regression, or it's extension mixture of generalized linear models that are all members of a wider family of finite mixture models, or latent class models.

It's not a combination of classification (supervised learning) and regression per se, but rather of clustering (unsupervised learning) and regression. The basic approach can be extended so that you predict the class membership using concomitant variables, what makes it even closer to what you are looking for. In fact, using latent class models for classification was described by Vermunt and Magidson (2003) who recommend it for such pourpose.

### Latent class regression

This approach is basically a finite mixture model (or latent class analysis) in form

$$f(y \mid x, \psi) = \sum^K_{k=1} \pi_k \, f_k(y \mid x, \vartheta_k)$$

where $\psi = (\boldsymbol{\pi}, \boldsymbol{\vartheta})$ is a vector of all parameters and $f_k$ are mixture components parametrized by $\vartheta_k$, and each component appears with latent proportions $\pi_k$. So the idea is that the distribution of your data is a mixture of $K$ components, each that can be described by a regression model $f_k$ appearing with probability $\pi_k$. Finite mixture models are very flexible in the choice of $f_k$ components and can be extended to other forms and mixtures of different classes of models (e.g. mixtures of factor analyzers).

### Predicting probability of class memberships based on concomitant variables

The simple latent class regression model can be extended to include concomitant variables that predict the class memberships (Dayton and Macready, 1998; see also: Linzer and Lewis, 2011; Grun and Leisch, 2008; McCutcheon, 1987; Hagenaars and McCutcheon, 2009), in such case the model becomes

$$f(y \mid x, w, \psi) = \sum^K_{k=1} \pi_k(w, \alpha) \, f_k(y \mid x, \vartheta_k)$$

where again $\psi$ is a vector of all parameters, but we include also concomitant variables $w$ and a function $\pi_k(w, \alpha)$ (e.g. logistic) that is used to predict the latent proportions based on the concomitant variables. So you can first predict the probability of class memberships and estimate the cluster-wise regression within a single model.

### Pros and cons

What is nice about it, is that it is a model-based clustering technique, what means that you fit models to your data, and such models can be compared using different methods for model comparison (likelihood-ratio tests, BIC, AIC etc.), so the choice of final model is not that subjective as with cluster analysis in general. Braking the problem into two independent problems of clustering and then applying regression can lead to biased results and estimating everything within a single model enables you to use your data more efficiently.

The downside is that you need to make a number of assumptions about your model and have some thought about it, so it's not a black-box method that will simply take the data and return some result without bothering you about it. With noisy data and complicated models you can also have model identifability issues. Also since such models are not that popular, there are not widely implemented (you can check great R packages flexmix and poLCA, as far as I know it is also implemented in SAS and Mplus to some extent), what makes you software-dependent.

### Example

Below you can see example of such model from flexmix library (Leisch, 2004; Grun and Leisch, 2008) vignette fitting mixture of two regression models to made-up data.

library("flexmix")
data("NPreg")
m1 <- flexmix(yn ~ x + I(x^2), data = NPreg, k = 2)
summary(m1)
##
## Call:
## flexmix(formula = yn ~ x + I(x^2), data = NPreg, k = 2)
##
##        prior size post>0 ratio
## Comp.1 0.506  100    141 0.709
## Comp.2 0.494  100    145 0.690
##
## 'log Lik.' -642.5452 (df=9)
## AIC: 1303.09   BIC: 1332.775
parameters(m1, component = 1)
##                      Comp.1
## coef.(Intercept) 14.7171662
## coef.x            9.8458171
## coef.I(x^2)      -0.9682602
## sigma             3.4808332
parameters(m1, component = 2)
##                       Comp.2
## coef.(Intercept) -0.20910955
## coef.x            4.81646040
## coef.I(x^2)       0.03629501
## sigma             3.47505076


It is visualized on the following plots (points shapes are the true classes, colors are the classifications).

For further details you can check the following books and papers:

Wedel, M. and DeSarbo, W.S. (1995). A Mixture Likelihood Approach for Generalized Linear Models. Journal of Classification , 12, 21–55.

Wedel, M. and Kamakura, W.A. (2001). Market Segmentation – Conceptual and Methodological Foundations. Kluwer Academic Publishers.

Leisch, F. (2004). Flexmix: A general framework for finite mixture models and latent glass regression in R. Journal of Statistical Software, 11(8), 1-18.

Grun, B. and Leisch, F. (2008). FlexMix version 2: finite mixtures with concomitant variables and varying and constant parameters. Journal of Statistical Software, 28(1), 1-35.

McLachlan, G. and Peel, D. (2000). Finite Mixture Models. John Wiley & Sons.

Dayton, C.M. and Macready, G.B. (1988). Concomitant-Variable Latent-Class Models. Journal of the American Statistical Association, 83(401), 173-178.

Linzer, D.A. and Lewis, J.B. (2011). poLCA: An R package for polytomous variable latent class analysis. Journal of Statistical Software, 42(10), 1-29.

McCutcheon, A.L. (1987). Latent Class Analysis. Sage.

Hagenaars J.A. and McCutcheon, A.L. (2009). Applied Latent Class Analysis. Cambridge University Press.

Vermunt, J.K., and Magidson, J. (2003). Latent class models for classification. Computational Statistics & Data Analysis, 41(3), 531-537.

Grün, B. and Leisch, F. (2007). Applications of finite mixtures of regression models. flexmix package vignette.

Grün, B., & Leisch, F. (2007). Fitting finite mixtures of generalized linear regressions in R. Computational Statistics & Data Analysis, 51(11), 5247-5252.

• When is this actually used in practice, what assumptions do you need to make about your model, and do you need to have rough upper/lower bounds on the number of classes, e.g. "between k=2-4" vs "between k=300-1000"? How does the performance vary with parameter k, is it O(k log k), O(k^2) or worse?
– smci
Jul 26, 2020 at 6:00

Multi-Task Learning MLT allows different types of loss-functions ( for example, least-square for regression and logistic or Hinge loss for classification) to be optimized simultaneously. the components of this heterogeneous loss function can be weighted to control/ distinguish the main task from the secondary one. if the two tasks do not have the same learning difficulties and convergence rates; a stop criterion has to be introduced for the simpler task to avoid overfitting. a 3rd component can also be introduced to the loss-function to ensure smoothness of the whole learning process. the heterogeneous loss function may look like that (a case for regression and classification):

notice the applied weight to the logistic loss function, and the last regularization term for eights penalization

Now if we want to implement this with Pytorch, we have to split the output and run it through different criteria (again MSE for regression and logistic loss for classification)

let yhat the inintial output of the model that is splited into yhat_1 and yhat_2 such :
yhat = concat(yhat_1, yhat_2)

the same for y the ground truth. in the learning step the model should be optimized as follow:

criterion1 = nn.MSELoss()
criterion2 = nn.BCELoss()
loss1 = criterion1(yhat_1, y1)
loss2 = criterion1(yhat_2, y2)
loss = loss1 + lambda*loss2
loss.backward()

• It's hard to understand why something as complicated as these solutions is needed. It's best to analyze the rawest form of the data. Classification (if really needed; a case was not made for it) can be done on the basis of the continuous predictions once the utility function is defined. Nov 16, 2020 at 12:11
• I have to detect facial landmarks from face images and classify each detected landmark as occluded or not occluded. so landmark detection is a regression problem that maps the image space to the landmarks coordinate space and the landmark classification is obviously a classification problem.
– yuri
Nov 16, 2020 at 13:13
• No it's not obvious that it is a classification problem, and forced-choice classification is misleading when there is a gray zone, i.e., intermediate probability of class membership that should result in "no decision, get more data". Once you develop optimum prediction you can combine that with a utility function to get an optimum decision, avoiding the need to classify. Nov 16, 2020 at 13:28
• how would this ''utility function'' look like? the dataset was labeled with landmarks coordinates and 0/1 occlusion status. the dataset supplier is assuming landmark classification. the link that you provided isn't directly treating ML for image analysis. but I should confess one can learn a lot from it.
– yuri
Nov 17, 2020 at 8:04
• For some tasks, reporting continuous predicted values or estimated probabilities of class membership is all that's needed, and you defer the decision to a decision maker who has her own internal utiities. Otherwise if you define a utility/cost/loss function e.g. quantify the relative harms of classifying a true category A as a B and of classifying a true category B as an A you can pick the category that minimizes expected harm/cost, i.e., that maximizes expected utility. Nov 17, 2020 at 11:42