Difference between semi-supervised learning and prediction?

Question

What is the difference between semi-supervised learning and prediction? It seems to me they're the same (both are predicting the label)

Answer 1

I will assume that you are talking about classification. Your question highly implies this, even if you don't use the word classification explicitly. Many algorithms in semi-supervised learning are transductive as opposed to supervised learning which is almost always inductive. This means that they cannot be used in the same way to predict new observations. Let me explain:

Inductive means that once you have learned a model based on your training set, you can predict unseen examples. Those could be a test-set which you put aside, but it could also be new records that trickle in over time (as they would in many real world applications). (Transductive supervised algorithms are very rare, so we can ignore them for now.)

Transductive semi-supervised algorithms differ in two ways from supervised classification:

1. They can use unlabeled examples in the training set (hence semi-supervised). Supervised algorithms would need to ignore those unlabeled records.
2. They can only predict those unlabeled examples (hence transductive). They will not create a model which can then be used to label new unlabeled records once they arrive (except if you restart the whole algorithm from scratch of course).

The reason why semi-supervised algorithms are transductive is inherent in their mathematical structure. For example, there are graph-based semi-supervised algorithms who do the following:

1. Find for each record (labeled or not) its k nearest neighbors.
2. Make a graph where each node is a record and there is a connection between neighboring nodes as determined in 1. The weight of these connections can be specified in a number of ways.
3. Propagate the labels over the constructed graph from labeled to unlabeled nodes.

You can see that once the graph is constructed, you cannot very easily insert new nodes into it. One problem here is step 2., the weights of the connections are optimized in sophisticated ways and usually all at once. Adding a node somewhere would affect the other weights in the graph or conversely, the only way to compute the weights that connect the new node is to recompute all weights. Even if that problem could be solved, step 3. can be computationally expensive and would still need to be repeated every time you add a node to the graph, so you might instead just relaunch all the steps.

I wrote my master thesis about inductive extensions to transductive semi-supervised graph-based classification algorithms. There are some ways to do it, but it's not easy. I can put academic references when I have a little more time.

On the other hand, supervised algorithms can use the labeled records to learn a model and then predict the labels of the unlabeled records, that were available form the start, using that model. So yes, you can use them to tackle the same task a transductive semi-supervised algorithms can do. However, they cannot use the information in the unlabeled records when predicting them. Since data-sets can have 5% or less labeled records, that means you are ignoring a lot of information when predicting. The assumption is made that using the >95% percent of unlabeled data for prediction is better than ignoring it when learning the model. Hence the need for semi-supervised algorithms. This assumption has seldom been challenged. In my master thesis however, I found that it may well be preferable to just ignore the unlabeled data when constructing the model, at least when the alternative would be a graph-based semi-supervised algorithm.

Answer 2

The difference is that prediction algorithms, by which I assume you are referring to supervised algorithms, have two stages: training and inference, but semi-supervised train and perform inference at the same time.

At training, supervised algorithms modify parameters to fit the data, but at test time - at inference - they simply perform prediction without modifying underline parameters.

Semi-supervised algorithms does not need separate stages, they could learn on the fly which is they perform inference and use the result of the inference to learn the parameters. As a side effect, these could be applied online. A good example where such algorithms are applied are in visual tracking where in the new video the object being tracked is different from whatever the algorithm have seen before so it has to adapt parameters on the fly.

Answer 3

Semi-supervised learning and prediction are two different concepts. Prediction is a task and semi-supervised learning is a learning scenario. Learning scenario used to describe how to training & testing data arrived and used. For semi-supervised learning, which means we get labels for training data but not very clear. But prediction is a task, which can combined with different learning scenario. Like we can get supervised prediction, unsupervised prediction and semi-supervised prediction.