Regression problem with “included classification”

I have basic knowledge on machine learning techniques and I'm working on a real-world regression problem which is basically predicting the time consumption of a certain task based on a number of predictor values. Unfortunately (as usual) the underlying relationship is not known to me. However, I do assume that there are different types of tasks in my data and therefore I'm searching for a technique to handle this.

Breaking it down to a simpler regression problem: I have one predictor p1, a categorial predictor p2, and the resulting time t, and my data looks looked like this: The regression becomes rather easy, as long as p2 is used by the regression model.

My Questions are:

1. Is there any type of machine learning technique for regression that does this kind of "included classification"? What I means is that it automatically builds regression models for the seperate classes, if that improves the model.
2. Would that technique work if the second predictor would not be categorial but "Type A" had p2 between 10 and 20, and "Type B" had p2 values between 30 and 40?
3. What are good approaches to handling categorial predictors in regression?
4. Any other advice for my described real-world problem?

Thanks for your help

//Edit 17.12.: Removed Example without p2

Assuming that you don't really have the categorical labels available, you can use an expectation-maximization approach to estimate iteratively (1) the labels of each observation and (2) the regression equation given each label.

If you do have the categorical labels available, for each label produce a predictor that equals p1 for observations that belong to that label and zeros elsewhere. If you have two labels, then it's two predictors. Then you solve it as a multiple regression problem. If you have continuous second predictor (p2) instead of a categorical one, then you can cast it as a problem of multiple regression with an interaction term: y = b1*p1+b2*p2+b3*p1*p2 .

• Thanks for your answer. Your last comment on the interaction term was very helpful. Are there any non-linear models that include such interaction terms automatically? – Felix Dec 17 '13 at 10:30

There is no "included classification" problem, you either have the features to properly predict your target value or not. To continue with your simplified example, if you had only "p1" the best you can do is use a model that can return a range of the possible values (since there are 2 distributions there, and you don't have enough information in your features).

If you have p2, just do normal regression. There is no "trick" in particular.

For your example, the p1 case is one where the distributions overlap and you lack the information needed to distinguish between them. It is not possible to create that information from nothing. You need to get more features for your data, thats simply the end of it. You can't extract something out of nothing.

The alternative case is where you have 2 distributions that do not overlap (i.e.: your features can disambiguate between the underlying distributions). In such a case you simply need to use a model / create a model that can handle them together.

Finally, there can be a mix - where there is some moderate amount of overlap between two underlying distributions. It happens. Again, pick the model that gets the most of it right / is most appropriate for your data.

• I'm well aware that the problem is not properly solvable without p2. But my questions were specifically aimed at which regression methods that are able to detect and handle different "distributions" within the data. Any comments on this? – Felix Dec 17 '13 at 10:17
• Thats now how the questions comes off. And all regression methods are meant for this - thats the point of regression. Its your job to figure out which one is appropriate for your data. No one can know which will work best without trying it themselves. – Raff.Edward Dec 17 '13 at 15:03

The expectation-maximisation suggestion by Tal is a very good place to start.

Another thing you might wish to try is Random Sample Consensus (RANSAC). This is primarily described as a method for finding patterns against high background noise. However it will also work in the situation you illustrate above, where there are multiple (possibly overlapping) instances of an otherwise simple pattern. It's used in the computer vision field for this kind of task (i.e. finding zero, one or more instances of a set of learned visual representations of objects, usually against a never before seen background) quite frequently.

On a related note, for simple schemes like linear regression, look at Hough transforms.