I am starting my adventure with machine learning and there is one concept I don't understand, so:
How one does determine classifier accuracy in real life machine learning system?
Let me give you an example I have in mind - lets say I have classifier ensemble consisting of 3 classifiers, data come in chunks. The pruning criterion is somehow correlated to the accuracy of each classifier in the ensemble (for the sake of this example lets just say that in every iteration the "weakest" model is dropped and replaced by another classifier from the pool). In "laboratory environment" I can easily check the accuracy of each classifier, because I know the class of the every object in the data set (assuming it is supervised learning) - but in the real life system I don't have that kind of information. The only class I have for given object is the one predicted by the classifier, so how do I determine how "strong" each model is so I can drop the weakest one?
This question might be trivial, but I would appreciate some explanation on this.