2
$\begingroup$

I am working with 12 different linear regression (e.g. Ordinary least square, Quantile, Huber regression and so on).

I calculated a performance measure for each linear regression, and I do this for 20 datasets (a table 20 by 12. twenty rows to the name of a dataset and twelve columns for the linear regression models and the table cells for the performance measure values).

I want to Calculate pairwise comparisons (To test every two pairs) to see if the performance measure is significantly different or not.

My question is

Does this problem need Independent or Paired Test? parametric or nonparametric test?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Independent and paired tests differentiate between statistical tests that compare two groups. However, you are comparing twelve groups and will need a statistical test that can handle this.

I'm assuming you have 12 "groups" (linear models fit by different regression algorithms) and for these groups you have "data" generated by performance of fitting on 20 different data sets. If normality and equal variance assumptions hold you can do a one-way ANOVA, and if not you can use Kruskal-Wallis. This will determine if model performance is significantly different between groups.

If the ANOVA or Kruskal-Wallis indicate that there are differences, you could use a paired t-test to compare the data. You may want to make sure to have corrections for post-hoc analysis to prevent false positives. See this question regarding post-hoc t-tests after rejection of ANOVA null hypothesis.

Improve this answer
$\endgroup$
1
  • $\begingroup$ My answer is the same. $\endgroup$
    – Alex Van de Kleut
    May 7, 2020 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.