# How to use data analysis output (e.g. clustering) in predictive regression?

I performed some data analysis and visualizations on my dataset and found there are likely $k$ clusters present. How can I use this in a predictive regression setting?

My first thought is to create a regression model within each cluster. Therefore given a testing $x$, decide what cluster it belongs to, and perform the predictive regression within that cluster. Note that the regression is completely independent of the data in other clusters. But this seems like there is data inefficiency, especially worse if the size of the cluster is not very large.

On the other hand, the performance of a global regression model is not necessarily affected by the presence of clusters, so I can ignore the clusters. But this seems like a waste of knowledge.

TLDR: How can I use clusters in regression?

• Any idea what causes the clustering? – cbeleites Dec 15 '12 at 12:01
• In general, no. – Cam.Davidson.Pilon Dec 15 '12 at 15:26
• What sort of data do you have, and what are the clusters based upon? – jbowman Dec 15 '12 at 20:41
• I'm more interested in a general answer, not so data specific. the question was inspired by the boston housing prices dataset fyi. – Cam.Davidson.Pilon Dec 15 '12 at 23:06

The shortest answer: it depends. Appearance of several clusters in the data is a strong hint of several data-generating processes in play. It is quite possible that they have different error term properties. Disagree with @Zach, since cluster classification adds no new information into the regression, and the differences in effects would simply be dumped into the "cluster" indicator. The first question to ask would in my opinion be: are the coefficients in the cluster-specific regressions significantly different from each other (or from the pooled regression)?

Especially if you are severely lacking degrees of freedom, pooled regression looks much more promising.

So far we have been talking about in-sample properties of your set, but as long as you want to use it for prediction, it is out-of-sample performance that should ultimately drive your decision-making.

• cluster classification can add new information to the regrssion, I think. If you run, say, a random forest to classify but a linear regression to regress, then clearly the random forest could split the data in non-linear ways that are precluded to the linear regression. Am I making sense here? – CarrKnight Mar 7 '13 at 18:12

I totally agree with Deer Hunter that we don't have enough information to answer your question. Moreover, an answer will usually not be possible from within the data, it will depend on the application and data-generating processes.

For predictive models, ultimately the validation with independent representative test cases will show whether what you did was a good idea or not.

The fact that you have clusters means that you have to be careful if using resampling validation schemes (cross validation or out-of-bootstrap). However, the examples below show that the apparent clusters may not coincide with the important structures in the data generating process. So knowledge of clusters only tells you that you'll probably run into trouble with "naive" resampling of the cases, but you cannot conclude about how the appropriate regression and resampling set-ups. Therefore, an inner loop of resampling validation may not be able to guide your modeling.

Some situations from chemical calibration with different clustering as "symptom" and conclusions for the appropriate regression model differing independent of the type of observed clustering. In all examples, the concentration of some analyte is to be modeled.

Situation 1: A number of calibration samples is obtained. From each of these calibration samples, a number of measurements is taken.

The reference value may be pre-specified (calibration samples are mixed accordingly), or samples are taken e.g. from a production process and reference values are measured with some other method.

You expect a strong clustering because of the repeated measurements, but per-cluster-regression is utterly meaningless. However, this situation is unproblematic, as you'd immediate notice that the information for the dependent variable is between the clusters, not within.

Situation 2: Calibration samples are often prepared by dilution of a stock solution (German Wiki on serial dilution is much better than the English. Dilution series of several independently prepared stock solutions are measured.

Per-stock-regression may lead to better prediction of measurements from the same stock. However, this is usually not appropriate, as rarely samples from the same stock are to be predicted.

You may see clusters in the independent data like in situation 1 that come from the fact that concentration series are usually prepared at "pretty" levels.

You may see the different stocks as clusters in the residuals (if instrument noise and random error in arithmetic dilution is << random error in concentration of stock solutions -- usually a bad sign wrt. the preparation).

Situation 3: You run into matrix effects and your data comprises a number of different matrices. In this case, per-matrix-regression is a valid option an can be much more successful than an overall regression.

Like in situation 2, you may encounter systematic patterns in the residuals of an overall regression.

Actually, you'd need to state for each regression model exactly in which situation it is appropriate. Your models cannot be expected to generalize to unknown matrices.

You could add them in as a categorical variable.