It is considered bad practice to pick the significance level (or $\alpha$) post-simulation.
Two reasons for picking the confidence level beforehand:
- The significance level is one criterion often used in deciding on an appropriate sample size. See e. g. here.
- The analyst is not tempted to choose a cut-off on the basis of what he or she hopes is true. Source
Jim Frost phrases the second point nicely: "It protects you from choosing a significance level because it conveniently gives you significant results!". For a graphical example and further elaboration see his post.
So reporting the p-values for relevant parameters makes sense and should be done (after picking the significance level for your study/case). But just always adding them "because you can" doesn't make sense, consider what information is gained from the reported parameters.
Here some more background on significance levels and reasons to consider them carefully for each case:
"No scientific worker has a fixed level of significance at which from
year to year, and in all circumstances, he rejects hypotheses; he
rather gives his mind to each particular case in the light of his
evidence and his ideas.” - Fisher (1956 in Statistical Methods and Scientific Inference, p. 42)
The University of Texas in Austin has some nice webpages on the topic, from which I will quote:
It is important to consider the implications and possible consequences of Type I and Type II errors before beginning with data analysis. So it is also a consideration between practical statistical significance. Consider the following example for the difference between the two.
A large clinical trial is carried out to compare a new medical
treatment with a standard one. The statistical analysis shows a
statistically significant difference in lifespan when using the new
treatment compared to the old one. But the increase in lifespan is at
most three days, with average increase less than 24 hours, and with
poor quality of life during the period of extended life. Most people
would not consider the improvement practically significant.
Now back to Type I and Type II erors and why their consideration is so important. (Here a small recap on Type I and Type II errors from datasciencedojo.com:)
Again, this can probably best be explained with examples (also from Texas University).
- If the consequences of a type I error are serious or expensive, then a very small significance level is appropriate.
Example 1: Two drugs are being compared for effectiveness in treating
the same condition. Drug 1 is very affordable, but Drug 2 is extremely
expensive. The null hypothesis is "both drugs are equally effective,"
and the alternate is "Drug 2 is more effective than Drug 1." In this
situation, a Type I error would be deciding that Drug 2 is more
effective, when in fact it is no better than Drug 1, but would cost
the patient much more money. That would be undesirable from the
patient's perspective, so a small significance level is warranted.
- If the consequences of a Type I error are not very serious (and especially if a Type II error has serious consequences), then a larger
significance level is appropriate.
Example 2: Two drugs are known to be equally effective for a certain
condition. They are also each equally affordable. However, there is
some suspicion that Drug 2 causes a serious side-effect in some
patients, whereas Drug 1 has been used for decades with no reports of
the side effect. The null hypothesis is "the incidence of the side
effect in both drugs is the same", and the alternate is "the incidence
of the side effect in Drug 2 is greater than that in Drug 1." Falsely
rejecting the null hypothesis when it is in fact true (Type I error)
would have no great consequences for the consumer, but a Type II error
(i.e., failing to reject the null hypothesis when in fact the
alternate is true, which would result in deciding that Drug 2 is no
more harmful than Drug 1 when it is in fact more harmful) could have
serious consequences from a public health standpoint. So setting a
large significance level is appropriate.