Extrapolating from a filtered data set Imagine the following hypothetical machine learning for classifying benign/malignant cancer tumors.
The doctors want to minimize the number of patients they call in for tests. They had an original data set of size 10,000 but based on the features chose only to examine those they expected to have cancer; 1,000 patients.
If I train a machine M on the 1,000 data points (out of which only 100 are
yes),


*

*what do I know about M?

*what does M tell me about an instance from the original 10,000 data points?

*how well does M work on new data points that are not chosen by doctors?


Is the data set useless?
 A: You have a model that tells you $\newcommand{\malignant}{{\rm malignant}}\newcommand{\features}{{\rm features}}\newcommand{\other}{{\rm other}}\newcommand{\doctor}{{\rm doctor}}\newcommand{\False}{{\rm False}}\newcommand{\True}{{\rm True}} P(\malignant|\features, \doctor=\True)$, where $\features$ are whatever datapoints are in your dataset, and $\doctor$ is the doctor's prediction on whether or not cancer is likely.
You'd like to have $P(\malignant|\features)$. But in order to get that, you need $P(\malignant|\features, \doctor=\False)$, and a way to determine $\doctor$. (One easy way to do it is, well, just ask the doctor.) If you know who eventually got a malignant cancer out of your dataset of 10,000, then you're set, but it sounds like you don't have that (or won't, for a few years).
Where I would look next, when trying to solve this problem, is whether I can determine $P(\doctor|\features)$. One thing to be worried about here is the doctor making a decision based on information that doesn't make its way into $\features$--in the real world, the doctor is making a decision based on $P(\doctor|\features, \other)$. If the performance of your $P(\doctor|\features)$ model is approximately as good as your $P(\doctor|\features, \other)$ model, then that's evidence for the trustworthiness of your model in the general case.
(In such a case, it's likely reasonable that $\features$ captures all the important information, and so you can just use the $P(\malignant|\features, \doctor = \True)$ model regardless of the value of $\doctor$.)
There's no principled way to totally exclude the risk that $P(\malignant|\features, \doctor=\False)$ is meaningfully different from $P(\malignant|\features, \doctor=\True)$ besides experimentation on that branch of the tree. If $\other$ turns out to be meaningful, then it probably is the case that the two of them are different, and so the model trained on the narrow case should be only cautiously applied to the general case.
A: This sounds a bit like the specific case of doing mammography before the decision to perform a biopsy in the detection of breast cancer. The prevalence of breast cancer in women at age 50 and above varies by age so we can restrict our analysis to women in a narrow age range, say 50-55 where the annual incidence is around 0.2% (which we can take to be on the same order of magnitude as the population prevalence of detectable cancers.) In this age group a radiologist is doing a reasonably good job if the biopsy-positive rate is around 10%, and you would be trying to substitute a machine algorithm that could improve upon that performance measure. So the rate of "sufficiently suspicious" mammograms is 2%, while the rate of positive biopsies is only 0.2%.
You may want to search for "machine interpretation of mammograms". Doing so I have already found:
"Support Vector Machines for Differential
Prediction", Finn Kuusisto, et al:
http://pages.cs.wisc.edu/~hous21/papers/ECML14.pdf
"Impact of computer-aided detection prompts on the sensitivity and
specificity of screening mammography", P Taylor et al: 
http://www.journalslibrary.nihr.ac.uk/__data/assets/pdf_file/0007/64987/FullReport-hta9060.pdf
That second one is going to have many, perhaps hundreds, of citations since it is one in a series of "Health Technology Assessment" and it has a full description of the methodology issues involved.
A: Generally, if you train M on a filtered dataset, you cannot use M on unfiltered datasets. Any kinds of artefacts may appear because of such a setting.  For example, a chess program, trained on grandmasters' games, moved its queen to a center of board early.  The chess theory prohibits it.  Why the program did it?  Because grandmasters move the queen, contrary to the theoretic suggestion, only when it is well protected.  The model just catched "who moves his queen to a center, is likely to win".
