Dietterich's 5x2cv paired t-test for regression problems In 1998 Dietterich proposed an algorithm for comparing classification algorithms using a series of cross validated t-tests: 
Dietterich, Thomas G. "Approximate statistical tests for comparing supervised classification learning algorithms." Neural computation 10.7 (1998): 1895-1923.
Is this test applicable for comparing regression algorithms? If not, what should be used instead? 
 A: There are two answers. The first one is, YES YOU CAN! The second answer is You may not need it.
First answer. Yes, you can use the 5x2cv paired test for regression. The only issue is that for each of the 10 test sets, you need a SUMMARY of the error of the regression for that test set. So you may use average regression error, or similar measures. You will have 10 of those measures, for each algorithm and you can use a paired wilcoxon test or a paired t-test to evaluate the p-value.
The second answer centers on the fact that you need a summary measure of error. For classification one uses a summary measure such as accuracy, or error rate (or F1 or AUC or any other measure for a SET of examples). For regression you don't really need a summary measure - you have a measure of error for each sample in the test set! You can do a pairwise comparison of the error measures for each sample in the the test, not for each test set! In this case you just perform a pairwise t-test or a pairwise wilcoxon test on the sample error. That is it. There is no need for cross validation.
As an aside, why we do not do this for classification? The problem is that for classification a by sample measure of error is a binary measure - the algorithm classified this sample correctly or incorrectly. Thus we have a paired set of binary measures for both classifiers. The correct test to use here is the the McNemar test which is a very weak test (will likely give you a high p-value and thus not significant differences). But in the case of regression, you have a paired set of numbers (the sample error in regression) and you can use the much more stronger paired t-test or paired wilcoxon test!
