2
$\begingroup$

Is it a good idea to mix supervised and unsupervised learning together? I'm trying to predict some sales data given some data on hand, so I think a regression is the best way to go. However, I'm also not sure on what is important information or not. So I was thinking that I start off treating this as an unsupervised learning problem (maybe use clustering?) to try and weed out any unnecessary attributes and then treat the remaining data as a regression problem. Or am I approaching this problem completely wrong?

I'm pretty new to machine learning, so I'm sorry if this is a strange question to ask.

$\endgroup$
1
  • 1
    $\begingroup$ Probably if you described your data, your aims and your ideas in greater details you would get better answers since now your description is quite vague. Btw, from your description it does not seem to be very related but there is such thing as semi-supervised learning (en.wikipedia.org/wiki/Semi-supervised_learning) $\endgroup$ – Tim Oct 25 '15 at 9:50
1
$\begingroup$

I personally do not recommend mixing unsupervised and supervised learning techniques. The simple reason for that is that the label can only aid you in your learning problem.

For example in your case you want to do feature selection as you are unsure about what features are best. Well then how about supervised feature selection? There are plenty of supervised feature selection methods available, of which the simplest investigates the correlation between each feature and your label.

Have a look at https://en.wikipedia.org/wiki/Feature_selection#Main_principles to start.

In the end I do want to emphasise that unsupervised feature selection techniques such as clustering can produce nice results. However, as a general rule of thumb: more accurate information yields better performance.

$\endgroup$
2
  • 4
    $\begingroup$ I wholeheartedly disagree. Unsupervised methods can refine and augment the input data (although not in the way OP is suggesting). Unsupervised feature generation for instance can be very beneficial to subsequent supervised tasks. Unsupervised pre-training for instance, is a great way to train deep neural networks, because it exposes patterns in the data which would be too difficult to find in a supervised way. $\endgroup$ – Youloush Oct 26 '15 at 9:40
  • $\begingroup$ I agree with your point in the case of deep neural networks. Unsupervised pre-training is sort of the standard there to obtain good results. However given that OP is new to machine learning I provided him with an answer that applies for most basic techniques $\endgroup$ – Sjoerd Oct 26 '15 at 9:56
4
$\begingroup$

From a definitional sense, there is no such thing as "mixing unsupervised learning and supervised learning" since any problem for which you have target variables is by definition supervised learning. When you don't have target variables it's called unsupervised learning. (See the introduction from C. M. Bishop, Pattern recognition and machine learning, 1st ed. 20. Springer, Oct. 2006.)

This is complicated because any supervised learning algorithm can be adapted into an unsupervised learning algorithm by letting the targets be the inputs. E.g., autoencoders are unsupervised neural networks. And any unsupervised learning algorithm can be adapted into a supervised one by letting targets be inputs as was done for RBMs (figure 4 here: Hinton, G. E. (2007). To Recognize Shapes, First Learn to Generate images. Progress in Brain Research, 165, 535–547.)

But in general, I think there is a clear difference between what typical unsupervised learning algorithms do well, and what typical supervised learning algorithms do well. Unsupervised learning algorithms create features from inputs: sometimes called discovery. Supervised learning algorithms learn mappings from features to targets: sometimes called learning (it's a bad name!)

If you have targets, I think it's better to apply supervised learning techniques. Even Boltzmann machines have a final training phase using gradient descent.

But if your input features are in an inconvenient space, or are highly correlated (in some space) then transforming them using unsupervised learning is going to help a lot.

If both points apply — you have targets and your inputs are correlated — then you should combine both techniques.

$\endgroup$
6
  • $\begingroup$ This is being automatically flagged as low quality because it is so short. In truth, it is more of a comment than an answer. Could you expand it? What do you mean by"RBM", eg, restricted boltzman machine? How would that work? Hoe does it mix unsup & sup? $\endgroup$ – gung - Reinstate Monica Oct 25 '15 at 20:31
  • $\begingroup$ @gung: Expanded. $\endgroup$ – Neil G Oct 25 '15 at 20:55
  • $\begingroup$ @NeilG I agree that it's definitional. However, if one treats the problem as a series of models, e.g., using the simplest example from unsupervised PCA to supervised modeling as would be done in many exploratory data mining exercises, then based on those results, mixing is possible across the series or stages of the analysis. $\endgroup$ – Mike Hunter Oct 26 '15 at 11:29
  • $\begingroup$ @DJohnson Right. In my final paragraph, I thought I made it clear that you can combine unsupervised learning techniques with supervised ones. The Hinton paper I cited essentially does this. $\endgroup$ – Neil G Oct 26 '15 at 11:32
  • $\begingroup$ @NeilG Sorry. My lousy parsing of your full statement in relying on your opening sentence. $\endgroup$ – Mike Hunter Oct 26 '15 at 11:35
3
$\begingroup$

I have no problem mixing "supervised" and "unsupervised" techniques. What regression modeling assumptions are violated by doing so? For instance, by flipping the mode of the data matrix under analysis from cluster solutions to PCA yields another "unsupervised" technique -- which is like regression without a dependent variable. Many people prefer to run PCAs as an exploratory step intended to weed out redundancy among predictors. So, depending on how you use the results, it can provide insight into the variable selection process, contributing to a "supervised" predictive model.

$\endgroup$
1
  • 1
    $\begingroup$ (+1) There's also hierarchical clustering of the predictor variables, followed by replacing the variables in each cluster by their first principal component (see Frank Harrell's RMS site). $\endgroup$ – Scortchi - Reinstate Monica Oct 26 '15 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.