Is cross validation only for comparing tuning parameters? If I'm creating a model (in R with caret), and don't have any tuning parameters I want to compare, is there any use for cross validation? In other words, I wouldn't use cross validation to validate my model, right? It's just for finding the model?
 A: It is better to have a separate test set to evaluate the performance of the model. However, if there are too few observations to make a split between test and train set, you could use the PRESS statistic instead, by removing the observations one by one and evaluating the sum of squared differences from the predicted values, using the remaining observations.  
$$\text{PRESS} = \sum_{i=1}^n(y_i-\hat{y}_{-i})^2$$
This procedure is a form of cross-validation, so I assume it is what you are referring to.
A: The primary purpose of cross validation is measuring model performance. 
So, yes, you can use cross validation results for validation (or, to be more precise for verification) purposes. 
The use of cross validation estimates during model tuning is just another use you can make of model performance estimates. However, it is important to understand that once you've used your cross validation results to drive model tuning, they have become part of your model training and are in that sense training error estimates. So in that case for valdiation adn verification purposes you need to measure the performance of the final model - leading to the typical nested or split-into-3-sets plans.

Whether test set or cross validation are better depends on a number of circumstances, namely


*

*the number of cases you have available (for testing),

*the figure of merit you try to estimate (proportions need far more test cases than e.g. mean squared error or Brier's score)

*whether you can really achieve splitting into independent training and test sets (depends e.g. on whether your data has clusters of cases)


So,


*

*If the test set is produced by the same splitting strategy (e.g. random selection of data rows) as cross validation, it is worse: it is subject to all the pitfalls of getting dependence between train and test set that cross validation is subject to. In addition, fewer cases are tested, leading to higher random uncertainty (variance) on the results.

*Independent test sets are better than cross validation if


*

*you use them to ensure independence by the data generation process, e.g. acquiring the test data only after model training is finished, and

*you make sure this test set is large enough to have acceptably low variance on the performance estimate. 


