I am asking this question because there seems to be a lack of a definitive answer despite how many variations of this "how to compare models" question have been asked. (see links about below for proof) I hope because of these links, people will take this question seriously, and I will get a reply.
How do compare linear regression (numerical) and decision tree model (classification) with cross-validation for the same dataset and target variable?
For my specific case, I have a variable (WDI's CO2 Emissions) that could either be numerical or factor/polynomial (1). So I want to be able to compare the both the most basic classification and numerical regression models possible. However because I am still learning about data science, I do not know what consistent statistic would measure accuracy, precision, and recall for both regression and classification models. What statistic could I use to compare? Below are the links that I have found while trying to solve this problem:
(1) For the record I did double check this with domain experts. CO2 emissions often aggregated/discretized as a factor of known size of 25. Read McMichael, A. J., Campbell-Lendrum, D., Kovats, S., Edwards, S., Wilkinson, P., Wilson, T., ... & Schlesinger, M. (2004) Or Global climate change OR Mahlstein, I., & Knutti, R. (2010). Regional climate change patterns identified by cluster analysis. Climate dynamics, 35(4), 587-600.
The broad links:
These do answer the question in the broadsense. However, they don’t agree about metrics, statistics, or measurements that can be used to compare the two using validation (especially because they are not always about comparing the two same models).
- How to choose a predictive model after k-fold cross-validation? (does not list metrics)
- Using and interpreting $k$-fold cross validation for regression (suggests to use both mean squared error and explained variance score)
- Comparing two GLMs using cross validation (answers only for numerical models)
- How to compare the performance of two classification methods? (logistic regression and classification trees) (answers only for classification models)
- How to compare classification methods in terms of performance? (answer for classification methods but suggests using the Brier Score or logarithmic score)
- Performing Cross Validation to Compare Lasso and Other Regression Models in R (very confusing)
- Cross-validation for Comparing Clustering Techniques (suggests using the T-test or F-test)
The confusing and erroneous links:
Note: When I was reading the links, they were confusing because some people talk about using cross validation to measure performance or evaluate individual models while comparing models. They talked about side issues, but didn’t answer the root question about how to use validation methods to compare models.
- How to evaluate results of linear regression
- Does cross-validation on simple or multiple linear regression make sense?
- Why not using cross validation for estimating the error of a linear model?
- Cross Validation confusion
- Training on the full dataset after cross-validation?
- Should cross-validation be used to provide the final parameters, or just to compare models?
P.S. Overfitting, even when using numerical regression, is a known issue for this particular to the dataset that I am using, just like it’s a problem for European weather model datasets. However, this is outside the scope of my questions as I using cross validation to compare models not measure an individual model’s performance.
Related Unanswered Questions of the same type by other users:
- How to compare performance of regression and classification?
- How to pick the best model with cross validation?
- https://stats.stackexchange.com/questions/244725/compare-classification-methods-with-cross-validation
- How to compare linear regression and classification trees? [without measuring error]
- Classification trees - how to avoid "cherry picking"?
- How to compare classification methods in terms of performance?
- Should cross-validation to compare models be performed with the same partitions?
- https://stats.stackexchange.com/questions/226995/compare-statistical-vs-nonstatistical-forecast-using-cross-validation
- How compare more than two methods after cross validation
- Cross-validation for Comparing Clustering Techniques
- https://stats.stackexchange.com/questions/244003/conducting-hypothesis-test-to-compare-two-models-test-error-generated-by-cross
- Comparing predictors based on ROC AUC and cross-validation error
- How to compare predictive models after cross validation if you have different response units?
