Usually, when you train your model, you split your dataset into two:
Train set is there for your model, so it can learn from it.
After your model is trained, you will show it data it has not seen before and see how well it will behave.
And now comes the validation part.
You are interested in how your model behaves in general. How well it can predict your target values. Is it poor estimation? Is it good one? And what if your model just learnt the pattern in training set (=overfitting).
So you must do validation on training set:
- What is the learning error?
- You do cross validation to see how it really behaves and that it does not overfit
(check this question)
And you must do validation on test set:
- To be sure that your model just did not learn the train set because you want to catch the big picture, not copy your train set
When validating, you can use various measures depending on your situation:
Accuracy, Precision, Recall, ROC curve, AUC, F1, etc.