2
$\begingroup$

In my Statistics class, we just talked about Statistical tests. So far I’ve been understanding the material, okay, but now I’m very confused.

I get that Statistical tests are used in hypothesis testing and assumes a null hypothesis of no relationship or no difference between groups to determine whether the observed data fall outside of the range of values predicted by the null hypothesis. And they are used to determine whether a predictor variable has a statistically significant relationship with an outcome variable and estimate the difference between two or more groups.

I'm not clear on when to perform a statistical test and choosing the right statistical test for parametric and nonparametric tests?

Can someone help me figure out choosing the right statistical test, intuitively, and how all those different statistical tests are related?

(References are welcome. I was able to find the descriptions of each statistical test separately, but what I am interested in are the comparison, applications, and advantages.)

$\endgroup$
2
  • 3
    $\begingroup$ Any chance you can ask a more specific question? This is what a master’s degree in statistics covers (incompletely). $\endgroup$
    – Dave
    Commented Jun 2, 2020 at 11:44
  • 1
    $\begingroup$ I find this a good reference on choosing the correct test: rcompanion.org/handbook/D_03.html $\endgroup$
    – Dave2e
    Commented Jun 2, 2020 at 12:08

2 Answers 2

1
$\begingroup$

For intuition - imagine you are walking along a road in a forest and find 5 stones arranged in a line and ordered from smallest to highest. You are surprised to find such a constellation and start wondering: how likely is it for this order to occur without human intervention?

Here you can check various different "null" causes. One might be a hypothesis that wind placed the rocks in such a way. You want to obtain a p-value for this "null hypothesis" so you start placing various rocks on a road, wait for heavy winds, and check if the wind produced such a pattern. You repeat this experiment thousands of times and get a p-value: number of cases where wind, in your experiment, ordered the stones just as perfectly (or more) as you found them on the road. But you can also test for other possibilities. Like maybe that rain helped to order the stones. Here you repeat a similar experiment, but this time you wait for rain. Measure the numbers, get the p-value.

Choosing the right statistical tests boils down to choosing the null "cause" for your observation. In this example different tests would be: testing for rain and testing for wind.

Another detail is the order of stones. If you only want to check how often the wind can order stones from smallest to largest - you do a one tailed test. But you might also count reverse orders, where the stones get placed in decreasing order from largest to smallest.

So, as you see, in effect different tests answer different questions. When you choose one test - you ask one question, and if you choose another test you are asking a different question.


Now let's move on from the stones and briefly touch on parametric vs non-parametric. Say you measure the difference in height between girls and boys in a class. And you want to test the probability for the observed mean difference to occur in a case where there are no difference between those groups in the population.

But you have a problem here. There can be various different scenarios hiding under the "there is no difference in a population" statement, just like there were various alternative under "stones were ordered without human intervention". You have to choose more concretely.

One possibility is to do a non-parametric tests. Let's pick permutations. With the permutation test you shuffle the heights between the groups and measure the mean difference between boys and girls using these shuffled values. You repeat the shuffling thousands of times and deduce a p-value, as the number of shuffles that had higher difference compared to the one that was observed. But behind such a procedure - you chose a concrete null population, without, maybe, being aware. What you in effect stated is that your "null" population is only composed of the heights you observed in a sample. In other words, you made an assumption that your sample reflects the population perfectly. And this is one possible statistical test.

Another possibility is a parametric test. Let's forget the t-test for now and pick something more naive - you can state that each of those groups are drawn from the same normal distribution. If they are, the distribution of their sample differences should be centered around 0 and have a variance that is also deducible from your data. So you pick mean at 0, calculate the standard deviation for both groups, pool them, and now you have a parametric distribution to draw from. For the sake of simplicity, imagine you draw thousands of numbers from this distribution (thousand of mean differences under null of no difference) and deduce a p-value by counting how many of those drawn numbers exceeded the observed value. What did you do here? Instead of claiming that your sample reflects the population perfectly, you made a different assumption about the population - that it is normally distributed with mean 0 and some unknown variance. Then you computed this variance on your sample and used a "plug-in" rule to state that the population has exactly the same variance as the sample. This is a different way to obtain the "null population", and hence a different statistical test.


To summarise: under the hood all of statistical tests work in a similar way. Beneath each of them there is a "null population" lurking. And by selecting between different tests you select a different "null population". Which in itself leads to asking a different question about your observed data.

$\endgroup$
1
$\begingroup$

Do a google search on "flow chart for choosing statistical test" and you will find an overwhelming amount of attempts to make that an easy choice. They may or may not be helpful in memorizing tests, I have never found any of them usefull in determining what to test.

As a beginner, you start learning tests like you learn words in a foreign language: I have learned the t-test, so I have learned to compare means of one or two metric variables. I have now learned the chi-square test of independence, so I have learned to compare count values of two or more groups. I have no learned ANOVA, so I can now do to many groups, what the t-test did to two groups. I have now learned ANCOVA, so I can have continuous predictors as well as nominal ones. I have now learned Pearson correlation, so I I have learned to determine a linear connection between to metrix variables. Your teachers will ask only questions that can be answered with the tests you have learned so far and if you have difficulties remembering them, then the above mentioned flow charts may provide memory support.

Later in your way into statistics you will learn methods that are less strict in what they do as you can to some extend tailor them to do what you want (linear regression, nonlinear regression, bootstrapping, permutation tests,...).

You asked for intuition and really, you should develop some intution for scale levels (binary, nominal, ordinal, count data, metric, ...) which are usually taugth right at the beginning of classes, when you cannot yet understand their fundamental importance in test choice.

JMTC, Bernhard

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.