Picking cases to label for classification I'm working on a classification problem with the goal of diagnosing kidney diseases from clinical data.  For each patient, we have a large number of observations, and would like to determine whether a patient has a particular kidney disease.  In a minority of cases, the diagnosis is known, but usually it is not, so it seems like a positive/unlabeled classification task.
However, there's a twist: if we want, we can have an expert review the case and determine whether the disease is present or not.  This consumes time and other resources, so we don't want to do it for every patient, but we could do it for some.
One option would be to have the expert review a random sample of cases.  However, I wonder if there's a way to guide the process, so that the expert reviews the most informative cases that would provide the greatest increase in classifier accuracy.  We're open to any type of classifier.
Any suggestions for how to think about this problem?  Any methods or tools we should employ?
Also, how should we deal with our data as being not-quite-positive/unlabeled, since we actually have positive, negative and unlabeled cases.
If it matters, the prevalence of the disease varies depending on the particular disease we're looking at (there are several) but ranges from <1% to ~30% in our cohort.
 A: So your problem is that you have labeled data, and unlabeled data. Look at 1st answer: https://stackoverflow.com/questions/19170603/what-is-the-difference-between-labeled-and-unlabled-data :

There are many active areas of research in machine learning that are
  aimed at integrating unlabeled and labeled data to build better and
  more accurate models of the world. Semi-supervised learning attempts
  to combine unlabeled and labeled data (or, more generally, sets of
  unlabeled data where only some data points have labels) into
  integrated models.

So you have to google for semi-supervised learning. This is way of state-of-the-art.
My way(without reading about semi-supervised learning so much), is to do unsupervised learning on whole data, to get clusters of similar cases. Then use humans(doctors) to describe clusters - is this cluster kidney disease or not?. Then you have data for supervised learning. And you can learn whatever you want Bayesian/LinearRegression/SVM classifiers.
A: Semi-supervised learning methods may well be the way to go, but I don't know it.  A clustering approach to guiding doctors towards more useful cases to label (as @user1615070 suggests) also has some merits.  
Let me suggest a different strategy.  In logistic regression, unlike linear regression, much of the information exists within a narrower range of your predictor variables.  Consider this contrived plot:  

Note that the probability of 'success' goes from $.2$ at $x = -.73$ to $.8$ at $x = .73$, a range of less than $1.5$ on $x$.  It is within this range that you have the least information about the status of the true label for $y$.  
With this idea in mind, I would use an iterative approach to gathering more labeled data:  


*

*Fit a logistic regression model to the labeled data you have.  

*Determine where the probable status of $y$ is unclear.  

*Sample unlabeled cases in the region where you are likely to gain the most information and have your experts label them manually.  

*Rinse and repeat as necessary.  


Using this strategy, you should be able to converge on a reasonable model efficiently.  
Note that this method assumes you already have some labeled cases with decent coverage of the predictor space.  From your description, I gather this is true.  However, if it isn't, you need a step 0 to precede the above.  If you have no idea from prior research where the probability shifts from undiseased to diseased, you would want to sample on a grid to get an initial sense of the target location.  
