I'm working on a classification problem with the goal of diagnosing kidney diseases from clinical data. For each patient, we have a large number of observations, and would like to determine whether a patient has a particular kidney disease. In a minority of cases, the diagnosis is known, but usually it is not, so it seems like a positive/unlabeled classification task.

However, there's a twist: if we want, we can have an expert review the case and determine whether the disease is present or not. This consumes time and other resources, so we don't want to do it for every patient, but we could do it for some.

One option would be to have the expert review a random sample of cases. However, I wonder if there's a way to guide the process, so that the expert reviews the most informative cases that would provide the greatest increase in classifier accuracy. We're open to any type of classifier.

Any suggestions for how to think about this problem? Any methods or tools we should employ?

Also, how should we deal with our data as being not-quite-positive/unlabeled, since we actually have positive, negative and unlabeled cases.

If it matters, the prevalence of the disease varies depending on the particular disease we're looking at (there are several) but ranges from <1% to ~30% in our cohort.

  • $\begingroup$ You could use mixture models assuming your subjects are drawn from one of 2 subgroups in the population: diseased or not. If your response variable is say normally distributed (maybe some measure of kidney function over time) then you could fit a 2-component normal mixture distribution to the data where each group is modelled with a (for example) linear mixed effects model. You could then incorporate the expert by fitting 2 separate mixed models to only those cases reviewed (hence you know the 2 groups). $\endgroup$
    – dandar
    Commented Feb 11, 2015 at 18:50
  • $\begingroup$ The parameter estimates obtained could then be used to build prior distributions on the model paramters so that you could then estimate within a Bayesian framework the mixture model on the labelled and unlabelled data. In this way once the expert has done their classification, the classification of subjects to diseased/un-diseased would be model-based. The accuracy of this classification could of course be checked against the cases the expert has reviewed. $\endgroup$
    – dandar
    Commented Feb 11, 2015 at 18:51

2 Answers 2


So your problem is that you have labeled data, and unlabeled data. Look at 1st answer: https://stackoverflow.com/questions/19170603/what-is-the-difference-between-labeled-and-unlabled-data :

There are many active areas of research in machine learning that are aimed at integrating unlabeled and labeled data to build better and more accurate models of the world. Semi-supervised learning attempts to combine unlabeled and labeled data (or, more generally, sets of unlabeled data where only some data points have labels) into integrated models.

So you have to google for semi-supervised learning. This is way of state-of-the-art.

My way(without reading about semi-supervised learning so much), is to do unsupervised learning on whole data, to get clusters of similar cases. Then use humans(doctors) to describe clusters - is this cluster kidney disease or not?. Then you have data for supervised learning. And you can learn whatever you want Bayesian/LinearRegression/SVM classifiers.

  • $\begingroup$ Partitioning Around Medoids might be the right clustering algorithm for the job. I found success with it in a similar application, although I ended up using a very small number of clusters (4-8) relative to my sample (>1,000) $\endgroup$ Commented Feb 12, 2015 at 9:09

Semi-supervised learning methods may well be the way to go, but I don't know it. A clustering approach to guiding doctors towards more useful cases to label (as @user1615070 suggests) also has some merits.

Let me suggest a different strategy. In logistic regression, unlike linear regression, much of the information exists within a narrower range of your predictor variables. Consider this contrived plot:

enter image description here

Note that the probability of 'success' goes from $.2$ at $x = -.73$ to $.8$ at $x = .73$, a range of less than $1.5$ on $x$. It is within this range that you have the least information about the status of the true label for $y$.

With this idea in mind, I would use an iterative approach to gathering more labeled data:

  1. Fit a logistic regression model to the labeled data you have.
  2. Determine where the probable status of $y$ is unclear.
  3. Sample unlabeled cases in the region where you are likely to gain the most information and have your experts label them manually.
  4. Rinse and repeat as necessary.

Using this strategy, you should be able to converge on a reasonable model efficiently.

Note that this method assumes you already have some labeled cases with decent coverage of the predictor space. From your description, I gather this is true. However, if it isn't, you need a step 0 to precede the above. If you have no idea from prior research where the probability shifts from undiseased to diseased, you would want to sample on a grid to get an initial sense of the target location.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.