I don't understand three important issues :
(1) Why do we calculate the test statistic under the null hypothesis ?
(2) Why do we reject the null hypothesis when :
p-value < significance level ?
(3) Why is type-I error more serious than type-II error ?
1)
I found this pre-print paper by @Michael Lew to clarify many things for me. In terms of calculating a p-value under the null hypothesis, it can be seen as more of a matter of convenience than anything else:
the null hypothesis serves as little more than an anchor for the calculation|a landmark in parameter space
To P or not to P: on the evidential nature of P-values and their place in scientific inference
2-3)
The original use of "significance" by Fisher grouped results into interesting/not interesting. If the p-value was not less than some threshold (unique to the given experiment determined subjectively) he would attribute it to low signal to noise ratio and ignore the result. There was no concept of type II error as one would never accept a hypothesis.
The Neyman-Pearson hypothesis testing framework instead determines the significance level via cost benefit analysis (what are the relative costs of Type I and Type II errors) using two (or more) plausible alternative hypotheses. The two approaches were combined into one by textbook writers starting in the 1940s and many claim the result makes no sense.
The wikipedia page on NHST needs some cleaning up to incorporate the contributions of different editors but it has many good references. Because the method that is taught is a result of historical accident (not reasoning or logic) it will be difficult for you to understand without knowledge of the history.
EDIT:
If we look at Fisher's early (first?) description of the significance test we see that in the case of the lady tasting tea it was plausible that she could not distinguish the order in which tea/milk was added at all. Thus this was a logical null hypothesis. He also states the significance level is "open to the experimenter".
However later he states that a null hypothesis that the death rates of two groups of animals are equal is also valid. In this case I think he simply missed the logical consequences of choosing an (often implausible; it is highly unlikely two death rates will be exactly the same) null hypothesis that is the opposite of the research hypothesis (why do the study unless you suspect the death rates will be different). These issues are described by Paul Meehl in Theory-Testing in Psychology and Physics: A Methodological Paradox and Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles That Warrant It
Quoting Fisher:
A LADY declares that by tasting a cup of tea made with milk she can discriminate whether the milk or the tea infusion was first added to the cup. We will consider the problem of designing an experiment by means of which this assertion can be tested.
...
It is open to the experimenter to be more or less exacting in respect to the smallness of the probability he would require before he would be willing to admit that his observations have demonstrated a positive result. It is obvious that an experiment would be useless of which no possible result would satisfy him. Thus, if he wishes to ignore results having probabilities as high as 1 in 20 -the probabilities being of course reckoned from the hypothesis that the phenomenon to be demonstrated is in fact absent- then it would be useless for him to experiment with only 3 cups of tea of each kind. For 3 objects can be chosen out of 6 in only 20 ways, and therefore complete success in the test would be achieved without sensory discrimination, i.e., by “pure chance,” in an average of 5 trials out of 100. It is usual and convenient for experimenters to take 5 per cent. as a standard level of significance, in the sense that they are prepared to ignore all results which fail to reach this standard, and, by this means, to eliminate from further discussion the greater part of the fluctuations which chance causes have introduced into their experimental results. No such selection can eliminate the whole of the possible effects of chance coincidence, and if we accept this convenient convention, and agree that an event which would occur by chance only once in 70 trials is decidedly “significant” in the statistical sense, we thereby admit that no isolated experiment, however significant in itself, can suffice for the experimental demonstration of any natural phenomenon; for the “one chance in a million” will undoubtedly occur, with no less and no more than its appropriate frequency, however surprised we may be that it should occur to us. In order to assert that a natural phenomenon is experimentally demonstrable we need, not an isolated record, but a reliable method of procedure. In relation to the test of significance, we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us a statistically significant result.
...
It is evident that the null hypothesis must be exact, that is free from vagueness and ambiguity, because it must supply the basis of the “problem of distribution,” of which the test of significance is the solution. A null hypothesis may, indeed, contain arbitrary elements, and in more complicated cases often does so: as, for example, if it should assert that the death-rates of two groups of animals are equal, without specifying what these death-rates usually are. In such cases it is evidently the equality rather than any particular values of the death-rates that the experiment is designed to test, and possibly to disprove.
I think a crucial point that others have missed in reading Fisher is that he never specifies a need to specify the significance level beforehand. A reasonable level could be chosen given the experimental conditions that unfolded. Also no strong conclusion about a natural phenomenon should be derived from a single study anyway, a single study could only provide evidence pointing towards the phenomenon's existence, thus suggesting the usefulness of further study. Thus the significance level could be arbitrary and based on convenience (since it could change) but it was not "binding".
While for Neyman-Pearson a single study could not provide any evidence for the existence of a phenomenon. The "significance" level was definitely non-arbitrary but it was binding if the pre-specified error rate of the decision making process were to be valid in the long run. Citing Neyman and Pearson the same as @MichaelLew in the above arxiv paper:
We are inclined to think that as far as a particular hypothesis is concerned, no test based upon the theory of probability can by itself provide any valuable evidence of the truth or falsehood of that hypothesis.
But we may look at the purpose of tests from another view-point. Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behaviour with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong.
The hybrid approach (which is being taught to researchers) uses a pre-specified arbitrary and binding significance level which is different from both. It neither adjusts the level due to experimental conditions nor involves cost-benefit analysis.
The test statistic is chosen to be a measure of the discordance of the data with the null hypothesis, in some direction of interest (e.g. the difference of sample means between two groups, the correlation between successive observations in time, &c.). The bigger it gets, the more evidence against the null hypothesis.
Well, we don't always. Sometimes the p-value is used just to calibrate the test statistic: a p-value of 0.03 means that you'd see a test statistic as big or bigger than the one you saw only 3 times out of a hundred if you were to keep repeating the measurement and if the null hypothesis were true. But if you want to say "I reject it" or "I don't reject it" (& to follow some different course of action in each case) then, when the null hypothesis is true, the p-value will be lower than a significance level 0.025 with probability 0.025 (it's uniformly distributed†); thus you control Type I error, the probability of rejecting the null hypothesis when it is in fact true, to 0.025.
More serious in what way? If you mean that making Type I errors has worse consequences, that's clearly not true in general.
† It's a bit different when you're sampling discrete variables—you'd have to say it'd be lower than the significance level with probability 0.025 or less.
