My main question is with regards trying to understand how k-fold cross-validation fits in the context of having training/validation/testing sets (if it fits at all in such context).
Usually, people speak of splitting the data into a training, validation and testing set - say at a ratio of 60/20/20 per Andrew Ng's course - whereby the validation set is used to identify optimal parameters for model training.
However, if one wanted to use k-fold cross-validation in hope of obtaining a more representative accuracy measure when the amount of data is relatively small, what does doing k-fold cross-validation entail exactly in this 60/20/20 split scenario?
For instance, would that mean that we'd actually combine the training and testing sets (80% of the data) and do k-fold cross validation on them to obtain our accuracy measure (effectively discarding with having an explicit 'testing set'? If so, which trained model do we use a) in production, and b) to use against validation set and identify optimal training parameters? For instance, one possible answer for a and b is perhaps to use the best-fold model.