Something like E-M for discriminative models? E-M provides a way to improve the estimation of a generative model with unannotated data. Is there anything out there that works the same way for discriminative models (e.g. perceptrons)?
For example, consider averaged perceptron tagger. It would be handy to be able to throw the entire Gigaword through some process of unsupervised model improvement.
EDIT:
So, I was pleasantly surprised to note that this site has the ambition of dealing with machine learning, but I'm learning by experiment what vocabulary is generic and what is very domain-specific. Apologies.
Consider a sequence classification problem, like part-of-speech tagging or named entity extraction. You can train a generative model (e.g. an HMM). That's a probability model, and you can apply E-M. However, the number of states grows prohibitive if you want to look at many features, and so the fashion tends toward things like CRFs (batch) or Perceptron (online).
For example, this paper talks about unsupervised learning in for a perceptron POS tagger, but the details are that they add the output of several pre-existing taggers to the training set of their model.
 A: This paper is very interesting, has good results, and is easy to apply to discriminative models.
And the term is semi-supervised learning, not unsupervised learning.
A: I think semi-supervised methods may be what you are looking for, there is quite a lot of litterature on this in Machine learning.  There is a good book on this topic, which gives a good idea of recent developments in this area.
An E.M.-like algorithm for logistic regression (a discriminative model) is easily implemented as follows:  
(i) Train a LR model using only the labelled data.  
(ii) Use the LR model to assign labels to the unlabelled data.  
(iii) Train a new LR model using both the labelled and unlabelled data (using the predicted labels).
(iv) repeat (ii) and (iii) until convergence is reached (i.e. none of the predicted labels for the unlabelled examples change).
This works quite well for some problems (e.g. text classification), not so well in others.  EM also works well with naieve Bayes.  McLachlan's excellent book on discriminant analysis also has some material on basic algorithms.
HTH
