How to set the significance level in a hypothesis test correctly

Question

I have difficulties understanding probabilities.

Let's imagine I have a test based on time series regressions and my null hypothesis is that I will pass my next exam in statistics. The test is performed on a small sample size and I choose a significance level of 5%. Unfortunately, I have to reject the null. :(

Luckily, I was educated by the best statisticians (in my city) and was told to be very conservative about doing such tests. Therefore, I decrease my alpha level to 1%, which in turn, decreases my rejection region and, hooray, now I fail to reject the null! I am going to pass! :)

Now, I also have a silly friend. Although she is a bright statistician, she is not very socially able. In her ingenuity she developed a statistical test herself based on cross sectional analysis that uses a large sample size. However, her null hypothesis is that I (=me) fail my next exam. Her initial choice of a a 5% significance level rejects the null hypothesis. Yet, because she is a nuisance to me, she lowers her significance level to 1% too with the consequence that she cannot reject the null either. Now, I am going to fail my exams!

So, according to my test at 1% significance level, I am going to pass. (I used a small sample size) According to her test at 1% significance level, I am going to fail. (She used a large sample size)

I have four emerging questions:

1) Who is right or wrong here?

2) If my friend and me had exactly the same sample size, can we be more sure if I am going to pass or fail my exam?

3) Does knowing the exact p levels help me to know more about who is right?

4) Given that we don't know anything about the structure of this two asymmetric tests, is it better to be against one own desirable outcome. In other words, if I want to pass, should I pick the test with the null hypothesis of failing - having my conservative education here in mind?

5) Is there a sure way to determine which test is better?

6) Am I going to fail my exam?

Answer 1

As explained in my answer in this post: What follows if we fail to reject the null hypothesis?, you can only find ''evidence'' for the alternative hypothesis $H_1$ i.e. when rejecting $H_0$.

The significance level is the probability of rejecting $H_0$ when it is true, so it is the probability of accepting $H_1$ when $H_0$ is true and by the above, the significance level is the probablity that you ''think'' that you found evidence while in ''reality'' it is false evidence.

So simplified; the significance level is the probability that you find ''false evidence''.

False evidence is ''scientifically'' bad, we think that we understand ''the world'' but our understanding is wrong.

So setting the significance level is up to you, it depends on how much you are willing to accept ''false evidence''.

Answer 2

Well, I have to say that there is not a correct answer to this because whatever the level of significance you always have Type I and II errors. This means that there is always a possibility of having false negatives and false positives. So it depends on the research that you are doing and in other things also.