0
$\begingroup$

I have two datasets consisted of 4 columns. I have to measure if the differences among the data (each column represent another data) are statistically significant. The problem is that:

  • in the first dataset we have 4 columns of errors commited by estimators (measured in milliseconds- delay of premature estimation). Each row is representing errors for one signal and each column represents different estimator (algorithm). Problem: the data (columns) are not normally distributed, they are usually skewed to the left side and some of them have big outliers and they have different standard deviations (first column has 5x smaller std. than the column which has the biggest std). Which test do I use in this case?
  • in the second case the dataset consists of 4 rankings represented in 4 columns and 10 rows (players). Each row consist of the score that player earned in particular ranking. There are some differences among these rankings for example- player who has been on the 7th place in the 1st ranking has fallen down onto 9th place in the second ranking (and so on). I want to confirm that these differences are statistically significant. According to KStest it is not normally distributed and again- the differences in standard deviations is large. Which test do I use in this case?
Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ Well, these are questions for Cross Validated. In your first question I would recommend starting with the Kruskal-Wallis test and Mann-Whitney U-test for post-hoc analysis. Both are beautifully implemented in R. $\endgroup$
    – Eli Korvigo
    Commented Jun 14, 2015 at 9:18
  • $\begingroup$ Thanks for your answer. Kruskal-Wallis requires normal distribution and Mann-Whitney is measuring '...especially that a particular population tends to have larger values than the other.'. In the first dataset it might be useful but probably after taking the absolute value. For the second dataset MW test is totally useless because rankings may have different values. The clue is to examine relations between rows. $\endgroup$
    – user2923339
    Commented Jun 14, 2015 at 11:29
  • 3
    $\begingroup$ Kruskal-Wallis doesn't require normal (or any) distribution. Its is a ranked test. Similarly, Mann-Whitney is a non-parametric and very powerful analog of t-test, hence it is a test of difference in means. It can be applied to your data in the first test without any additional manipulation, if you correctly described your data. $\endgroup$
    – Eli Korvigo
    Commented Jun 14, 2015 at 11:55

1 Answer 1

Reset to default
1
$\begingroup$

For the second question, the Friedman test compares multiple rank-orderings. Make sure you set it up so that you do the comparison you want: test whether there are there differences among rankings, as opposed to differences among players.

For the first question, the answer would depend on what you are trying to accomplish, in particular how you want to evaluate the estimators. Do you want to find the estimator that gives the lowest average error among signals, or that gives the lowest median error, or that gives the most reproducible error, or that is the least likely to make a really big error, or something else? If all of the data points have values greater than 0, a logarithmic transformation might help remove the skew and help you visualize the data better.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.