# Is there “unsupervised regression”?

If I am correct, "unsupervised classification" is same as clustering. Then is there "unsupervised regression"? Thanks!

I've never encountered this term before. I am unsure whether it would spread light or darkness within either realm of statistics: those being machine learning (where supervised and unsupervised distinctions are central to problem solving) and inferential statistics (where regression, confirmatory analysis, and NHSTs are most often employed).

Where those two philosophies overlap, the majority of regression and associated terminology is thrown around in a strictly supervised setting. However, I think many existing concepts in unsupervised learning are closely related to regression based approaches, especially when you naively iterate over each class or feature as an outcome and pool the results. An example of this is the PCA and bivariate correlation analysis. By applying best subset regression iteratively over a number of variables, you can do a very complex sort of network estimation, as is assumed in structural equation modeling (strictly in the EFA sense). This, to me, seems like an unsupervised learning problem with regression.

However, regression parameter estimates are not reflexive. For simple linear regression, regressing $Y$ upon $X$ will give you different results, different inference, and different estimates (not even inverse necessarily), than $X$ upon $Y$. In my mind, this lack of commutativity makes most naive regression applications ineligible for unsupervised learning problems.

• +1, and I vote darkness. A Google search yields a number of references to "unsupervised regression", many of which are of the structural equation modeling/latent classes flavor. From a brief review of these papers, I would personally tend to describe them as applying least squares (LS) and expectation maximization (EM) methods to unsupervised problems, rather than "unsupervised regression" – JBK Jun 10 '13 at 21:57
• Thanks! I wonder if unsupervised learning problems have commutativity? – Tim Jun 11 '13 at 16:11
• Most unsupervised learning applications I've encountered deal with covariance estimation and (highly related) clustering. Because in these applications you can arbitrarily permute columns of data without causing any distress, and there's no need to designate variables as either features or responses, I would say these applications are commutative. – AdamO Jun 11 '13 at 19:13

The closest thing I can think of is a little black magic that stirred people up when it was announced a few years ago, but I don't believe it gained any real traction in the community. The authors developed a statistic they called the "Maximal Information Coefficient (MIC)." The general idea behind their method is to take highly dimensional data, plot each variable against every other variable in pairs, and then apply an interesting window-binning algorithm to each plot (which calculates the MIC for those two variables) to determine if there is potentially a relationship between the two variables. The technique is supposed to be robust at identifying arbitrarily structured relationships, not just linear.

The technique targets pairs of variables, but I'm sure it could be extended to investigate multivariate relationships. The main problem would be that you'd have to run the technique on significantly more combinations of variables as you allow for permutations of more and more variables. I imagine it probably takes some time just with pairs: attempting to use this on even remotely high dimensional data and considering more complex relationships than pairs of variables would become intractable fast.

Reference the paper Detecting Novel Associations in Large Datasets (2011)

Auto regression is one way to compute weights of a matrix minimizing error on reconstructed input from given input.

This question came to my mind while researching the difference between supervised and unsupervised methods. Coming from an econometric background I prefer to think in models, which slowed my understanding as most machine learning literature I encountered focuses on methods.

What I have found thus far is that a strict distinction should be made between clustering (unsupervised) versus classification (supervised). The continuous analogy of the relation between these model designs would be principal component analysis (unsupervised) versus linear regression (supervised).

However, I would argue that the relation between clustering and classification is purely coincidental; it exists only when we interpret both model designs as describing a geometric relation, which I find unneccesarily restrictive. All unsupervised methods that I know of (k-means, elastic map algorithms such as kohonen/neural gas, DBSCAN, PCA) can also be interpreted as latent variable models. In the case of clustering methods, this would amount to viewing belonging to a cluster as being in a state, which can be coded as a latent variable model by introducing state dummies.

Given the interpretation as latent variable models, you are free to specify any, possibly nonlinear, model that describes your features in terms of continuous latent variables.