# What is the precise definition of unsupervised learning?

Let's look at a special case: Generative Adversarial Networks (GANs).

(For those who don't know what a GAN is: for this purpose they are two neural networks that are trained using user generated labels, that distinguish "fake" (generated) data and "real" (observed) data.)

I would argue that GANs are unsupervised (not CatGAN, or GANs that do classifications, obviously).

Wikipedia defines unsupervised learning (without any sources) as

Unsupervised machine learning is the machine learning task of inferring a function to describe hidden structure from "unlabeled" data (a classification or categorization is not included in the observations). Since the examples given to the learner are unlabeled, there is no evaluation of the accuracy of the structure that is output by the relevant algorithm [...]

However, MATLAB seem to have a different definition (on their home page):

Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labeled responses.

A GAN uses labeled response, and also there is an evaluation of the accuracy of the structure that is output.

So my question is: what is the most precise (or actual) definition of unsupervised learning?

TLDR; supervised vs unsupervised is just an classification, as any other classification, it is simplified and does not describe the reality perfectly, as in real-life nothing is black and white and you can always find areas of gray.

First, let's look at two definitions from the popular machine learning handbooks. In Pattern Recognition Bishop defines it as follows:

In other pattern recognition problems, the training data consists of a set of input vectors $\mathbf{x}$ without any corresponding target values. The goal in such unsupervised learning problems may be to discover groups of similar examples within the data, where it is called clustering, or to determine the distribution of data within the input space, known as density estimation, or to project the data from a high-dimensional space down to two or three dimensions for the purpose of visualization.

Hastie, Tibshirani and Friedman in The Elements of Statistical Learning define it as

In the unsupervised learning problem, we observe only the features and have no measurements of the outcome. Our task is rather to describe how the data are organized or clustered.

So basically, in supervised learning, you learn a function of your data, that lets you predict some labels (classes in classification, numeric values in regression). In unsupervised learning you don't have labels, you learn some kind of representation of the data from the data.

So when neural network learns how to re-create, or simulate an image, it is an unsupervised learning problem, as you are learning representation of the data. When given the data you try to classify or predict something, it is an supervised problem.

I am not an expert in neither deep learning, nor GANs, but what I know about them is coherent with what Wikipedia says

One network generates candidates and the other evaluates them. Typically, the generative network learns to map from a latent space to a particular data distribution of interest, while the discriminative network discriminates between instances from the true data distribution and candidates produced by the generator. The generative network's training objective is to increase the error rate of the discriminative network (i.e., "fool" the discriminator network by producing novel synthesised instances that appear to have come from the true data distribution).

so you can say that GANs do both: learn representation of the data, and given the labels of fake data, learn how to distinguish fake data from real. GANs use two networks that compete: one is unsupervised and the other supervised. Moreover, GANs are pretty special algorithm since they are designed to sample from the unobservable distribution of adversarial examples, so you can say that their ultimate goal is to learn the representation of the distribution (unsupervised problem).

• Well, to be pedantic, both the networks can be considered supervised, not only one of them. Although, you are training one of them (the generator) based on the output it gives when sent to the other (the discriminator). – Andreas Storvik Strauman May 29 '18 at 13:33
• @AndreasStorvikStrauman right, you may be right and this was oversimplification on my side, plus, it's not exactly my area of expertise. But, AFAIK, both networks are not exactly supervised. GANs are pretty special case, so I don't think the classification does any good in here. – Tim May 29 '18 at 19:52
• Classification using GANs are fine. Check out e.g. CatGAN :) – Andreas Storvik Strauman May 29 '18 at 20:46

Unsupervised learning is a procedure for understanding predictors and observations that in no way utilizes the variables you're predicting. Examples of understanding predictors are predictor variable clustering and principal components, and an example of understanding observations is observation clustering on predictors.

There is a special case: sliced inverse regression, where principal components are done on subsets of the data formed by values of the outcome variable. This special case still qualifies as unsupervised learning because principal components are intercept-free.

A method that uses only "fake" or randomly generated outcomes to estimate overfitting is not learning at all, but could reasonably be called unsuperised learning. It provides a way to estimate the bias due to overfitting that is inherent to the analytical strategy. Put another way, if you can get a decent $R^2$ from random data, you have a problem with your predictive algorithm.

A method that uses a mixture of "real" and "fake" outcomes is supervised learning.

• I'm not asking what unsupervised learning is and how it's used. This "definition" (your first sentence) is not clearing up what I ask about at all. Is my question unclear? – Andreas Storvik Strauman May 29 '18 at 11:58
• What? Look at the title of your question again. – Frank Harrell May 29 '18 at 12:05
• Yes, you give a definition of unsupervised learning, which is the answer to the title. However, I don't see how this helps in the GAN case? What would be the predictors and variables here? – Andreas Storvik Strauman May 29 '18 at 12:08
• I added to the answer. – Frank Harrell May 29 '18 at 12:17

Both definitions that you quoted are essentially the same. The idea's that in supervised learning you observe $x,y$, where $x$ are predictors and $y$ are responses. You try to map the inputs to outcomes as $y=f(x)$, i.e. try to extract mapping $f(.)$ from the observed data $(x,y)$.

In unsupervised learning you do not observe responses $y$. You try to extract both the unobserved responses $y$ and a mapping $f(.)$ from observed data $x$. Naturally, it's more difficult than supervised learning.

• The GAN consists of two functions that has trainable parameters. Both of these functions are conventionally trained using supervised methods. E.g. both of the functions (the generator and the discriminator) are networks, and both of them are trained on labels that you create! This is where my problem really lies - would you call the method an unsupervised learning model? – Andreas Storvik Strauman May 29 '18 at 13:31

The problem that GAN solves is unsupervised learning. There is no label only a set of unlabeled samples from the distribution (or all the samples have the same label). The method used to solve the approach is combining two supervised systems. The method is something different from the problem.