What is the precise definition of unsupervised learning?

Question

Let's look at a special case: Generative Adversarial Networks (GANs).

(For those who don't know what a GAN is: for this purpose they are two neural networks that are trained using user generated labels, that distinguish "fake" (generated) data and "real" (observed) data.)

I would argue that GANs are unsupervised (not CatGAN, or GANs that do classifications, obviously).

Wikipedia defines unsupervised learning (without any sources) as

Unsupervised machine learning is the machine learning task of inferring a function to describe hidden structure from "unlabeled" data (a classification or categorization is not included in the observations). Since the examples given to the learner are unlabeled, there is no evaluation of the accuracy of the structure that is output by the relevant algorithm [...]

However, MATLAB seem to have a different definition (on their home page):

Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labeled responses.

A GAN uses labeled response, and also there is an evaluation of the accuracy of the structure that is output.

So my question is: what is the most precise (or actual) definition of unsupervised learning?

Answer 1

TLDR; supervised vs unsupervised is just an classification, as any other classification, it is simplified and does not describe the reality perfectly, as in real-life nothing is black and white and you can always find areas of gray.

First, let's look at two definitions from the popular machine learning handbooks. In Pattern Recognition Bishop defines it as follows:

In other pattern recognition problems, the training data consists of a set of input vectors $\mathbf{x}$ without any corresponding target values. The goal in such unsupervised learning problems may be to discover groups of similar examples within the data, where it is called clustering, or to determine the distribution of data within the input space, known as density estimation, or to project the data from a high-dimensional space down to two or three dimensions for the purpose of visualization.

Hastie, Tibshirani and Friedman in The Elements of Statistical Learning define it as

In the unsupervised learning problem, we observe only the features and have no measurements of the outcome. Our task is rather to describe how the data are organized or clustered.

So basically, in supervised learning, you learn a function of your data, that lets you predict some labels (classes in classification, numeric values in regression). In unsupervised learning you don't have labels, you learn some kind of representation of the data from the data.

So when neural network learns how to re-create, or simulate an image, it is an unsupervised learning problem, as you are learning representation of the data. When given the data you try to classify or predict something, it is an supervised problem.

I am not an expert in neither deep learning, nor GANs, but what I know about them is coherent with what Wikipedia says

One network generates candidates and the other evaluates them. Typically, the generative network learns to map from a latent space to a particular data distribution of interest, while the discriminative network discriminates between instances from the true data distribution and candidates produced by the generator. The generative network's training objective is to increase the error rate of the discriminative network (i.e., "fool" the discriminator network by producing novel synthesised instances that appear to have come from the true data distribution).

so you can say that GANs do both: learn representation of the data, and given the labels of fake data, learn how to distinguish fake data from real. GANs use two networks that compete: one is unsupervised and the other supervised. Moreover, GANs are pretty special algorithm since they are designed to sample from the unobservable distribution of adversarial examples, so you can say that their ultimate goal is to learn the representation of the distribution (unsupervised problem).

Answer 2

Unsupervised learning is a procedure for understanding predictors and observations that in no way utilizes the variables you're predicting. Examples of understanding predictors are predictor variable clustering and principal components, and an example of understanding observations is observation clustering on predictors.

There is a special case: sliced inverse regression, where principal components are done on subsets of the data formed by values of the outcome variable. This special case still qualifies as unsupervised learning because principal components are intercept-free.

A method that uses only "fake" or randomly generated outcomes to estimate overfitting is not learning at all, but could reasonably be called unsuperised learning. It provides a way to estimate the bias due to overfitting that is inherent to the analytical strategy. Put another way, if you can get a decent $R^2$ from random data, you have a problem with your predictive algorithm.

A method that uses a mixture of "real" and "fake" outcomes is supervised learning.

Answer 3

Both definitions that you quoted are essentially the same. The idea's that in supervised learning you observe $x,y$, where $x$ are predictors and $y$ are responses. You try to map the inputs to outcomes as $y=f(x)$, i.e. try to extract mapping $f(.)$ from the observed data $(x,y)$.

In unsupervised learning you do not observe responses $y$. You try to extract both the unobserved responses $y$ and a mapping $f(.)$ from observed data $x$. Naturally, it's more difficult than supervised learning.

Answer 4

The problem that GAN solves is unsupervised learning. There is no label only a set of unlabeled samples from the distribution (or all the samples have the same label). The method used to solve the approach is combining two supervised systems. The method is something different from the problem.

Ps your title is misleading, you should add GAN there

Answer 5

Perhaps unsupervised problems are just supervised problems with $Y=X$. Consider some kind of abstract notion of projection?

We (rather wildly and waving hands) posit that any problem class $\mathcal{P}(\mathcal{Y})$ that a human $h$ labels as "unsupervised" can be formulated as problem class $\mathcal{P}(\mathcal{Y}, \mathcal{X})$ where the human $h$ would label them "supervised" but with $\mathcal{Y} = \mathcal{X}$.