Rookie question about non parametric and parametric tests

I am a very rookie statistical test user and I want to better understand how to choose between a non parametric and parametric test based on my data. I read several statistical tests tutorials but I couldn't make any correlation between the examples given and my data, so it was hard to understand the tutorials I read so far.

My data is composed by five vectors which represent ten f-measures values of a classification of a set of binary data. Each dimension of the vector represents the f-measure result of each 5x2 cross validation machine learning experiment.

My question is: how to choose between a parametric and non parametric test based on my data? also, how to know which test to use? is there any wrong option or it depends on the assumption I give to my data and I am free to decide?

The first evaluation should be based on the normality of the data -- that is, how bell-shaped a histogram of each feature looks like. Normally-distributed features result in a bell-shaped histogram with symmetric tails on the left and right. If you run a histogram to show the occurrence frequency over the range of say, one of your features, and it has a large left or right-skewed tail, then the feature is probably log-normally distributed. [if you take the log of the feature values in this case, then generate a histogram of log(x1), it might look more symmetric].

Given the above, "parametric" tests are based on parameters such as the average $\mu$ and variance $\sigma^2$, which are parameters of the normal distribution. Non-parametric tests are commonly based on ranks, and therefore order statistics.

For a simple example, say you have 4 values of a feature, which happen to be: 1,2,3,1000000. The average of these 4 values is a little above 250,000, but if ranks are first assigned, as in 1$\rightarrow$1, 2$\rightarrow$2, 3$\rightarrow$3, 1000000$\rightarrow$4, then average is 2.5. Overall, the use of ranks in non-parametric test prevents results from being biased by outliers, i.e., the value of 1000000 being next to small values of 1,2,3.

The difference is clear: parametric tests are based on parameters like $\mu$ and $\sigma$, which can be biased (incorrect) in the presence of outliers. However, non-parametric tests based on ranks of numbers do not use parameters such as the average and standard deviation, they simply use order statistics and expected values of ranked values.

The point of the above is that for the 4 example values provided, parametric tests would use the average value near 250,000, while non-parametric tests would have a much lower order statistic near the bulk of the data. In other words, a single very large (small) value for a feature can greatly throw off your test conclusions, while non-parametric tests (based on ranks) are not biased by large outlier values.

There are distributional issues involved as well, so the level of complexity increases the more you delve into parametric vs. non-parametric. But for the student, showing the different averages near 250,000 and 2.5 really drives home the idea of how non-parametric tests are less biased. (NB: non-parametric tests have less statistical power, if applied to data for which parametric tests are appropriate -- just a caveat).

• The normality of the overall variables does not matter, the normality assumption is on the regression residuals. – Björn May 29 '16 at 5:43
• Correct -- for regression the residuals need to be normally distributed, but the question was not focused on regression, it was focused on parametric tests vs. non-parametric. – JoleT May 29 '16 at 20:42
• @LEP so, suppose I want to use a non parametric test, what are the criteria should I use to choose the right statistical test on my data? – mad May 29 '16 at 22:54
• Your 5 vectors represent different classification methods? It appears as though you want to compare the average f-measure between the 5 vectors(?). Firstly, statistical tests would need to be employed to determine if any pair of the 5 vectors, have significantly different averages, which can be accomplished by using one-way ANOVA (parametric) or Kruskal-Wallis (non-parametric). Either way, just make a dataset with two variables: the first (x-categorical, or factor) taking on values 1,2,...,5 and the second with the f-measure (y-value). Then run ANOVA or KW using any stats package. – JoleT May 30 '16 at 0:17
• Let me know if you get in a bind -- for your data setup you will have 10 rows with a 1, followed by f-measures in the same rows in column 2, then rows 11-20 with a 2 in the first column, and f-measures in column 2, and so on, until all 50 observations are in the dataset. ANOVA requires a grouping variable (your factor variable 1) and a continuous variable (variable 2) with your f-measure. Running should be straightforward. – JoleT May 30 '16 at 0:22