To understand how successful/accurate the classifier training/learning was, do all your metrics on the data set which was actually presented and used by the classifier - 4000 examples in your case as I understand.
Precision and Recall typically are calculated for each class independently.
Also, you can combine different classes into one or split them in multiple based on your real-world interpretations and needs.
What you loose by focusing on just 4000 is the overall utility of the classifier on real world data. The classifier might work well on 4000/usable datasets but the fact is that it can handle only 80% of data the rest won't be usable by the classifier. This may or may not be of significance.
A doctor can help with general diseases(physician), lung related diseases and heart related diseases but is not a specialist in orthopedics(bone related troubles).
In this case patients with bone related problems will be turned down by the doctor(ideally) while to others the doctor will prescribe something. This is like 1000 getting rejected out of 5000.
Now out of the 4000 some might get good treatment while some bad which will tell you how good the doctor is in each class of disease for which he handles the patients (physician, lung problems, heart problems) if precision, recall and other metrics are calculated independently for each class on 4000
On 5000 if you calculate metrics like precision you will first have to create a class called turned down by doctor. And then find precision and recall for this (and others too) which should be 1 if doctor is able to identify and turn down orthopedic and diseases beyond his ken accurately. (in your world your classifier is able to turn down the data records accurately as you drop anything and only those things which don't have any of the feature/field needed). This precision and recall would represent how good the doctor is with all the patients where definition of being good is being good at :
- correctly prescribing to patients with known disease
- correctly turning down patients who have disease beyond his ken/expertise
Hope this helps