Measure precision This is used in information retrieval. 
I have an algorithm that uses a sample set to predict  a yes / no. 
I think the correct term is binomial. 
Yes this document is about sports or no this document is not about sports.  
I need to represent the effectiveness of the algorithm with that population and that question.
Can I just compare the output from algorithm to the sample set. 
Clearly I need remove that item from the sample set to fairly evaluate the prediction.
If I have a sample set of 1000 and as reviewed by an expert exactly 700 were about sports. 
If I go through the sample set one at a time and remove that item from the sample set and predict and compare the prediction to the actual value is that a fair measure of precision. 
If the prediction was correct on 990 of the items can I say the precision is 99%?
I think it is silly to have have to get another X samples to test the prediction. 
If I remove that item from the sample set then can I not pretend it was the test of the sample set / algorithm?
The sample set is random. I also use the sample set to estimate the number in the total population.
I get that if I tweak the algorithm for conformance to the sample set then I cannot use that as a measure of precision.  At that point I would need to test against a blind sample. 
 A: What you're doing is classification, and the metric you're using is accuracy, not precision. 
Accuracy is what percentage of times you predict correctly.
Precision is specific to one class, say the class of "about sports". Out of the times you say "yes it's sport", how often you are correct. So if you don't have any "false positives" (texts about cooking that you claim to be about sport) then your precision is 100%, even though you can make mistakes the other way (getting a text about sport and saying it isn't).
The complementary metric is recall.
The technique you're describing is called "leave one out" cross-validation: your "sample set" used as a basis for prediction is the training set, and you're removing one item at a time to use a a test set.
that technique is used when you have little data and it's cheap to retrain the classifier each time (because here it means you have to train 1000 times). 
You could also remove 10% of the training set and test against that, repeat 10 times with each time 90% of the data for training and 10% for testing, and that's the classic 10-fold cross validation.
Your final point about not tweaking your algorithm against the sample set is correct.
