Whether or not you keep the test set separate depends on what you want to do with the final model you build after running the K-fold cross validation.
If you use all the training/validation/test data in this process, you have no independent data to use for any further testing/evaluation. If you are happy with that, then use all the training/validation/test data in the cross-validation process; and use all the data to build your final model. In many situations this is fine, as you have the results of the cross-validation to assess model performance.
In some situations this may not be appropriate. For example, you may need to show that your final model performs as well as the cross validation results indicate it should. If that's the case, then if you don't use your test set for cross validation, you can use that to test your final model. The trade-off though, is this method may result in slightly lower-performing model, as you have used less data to train it.