About alfa and p Please, help me clarify something that is driving me crazy.
If the significance level alpha of my test say the "percentage" in which I might be "wrong" in finding that I can't reject the null hypothesis if the calculated p is greater than alpha, and if the calculated p says the "percentage" that the differences observed in the test are due to random variation, how come I can compare these two values? In my head alpha says one thing and p says another, so to me it seems like you are comparing apples and oranges.
 A: To avoid unnecessary 'fuzziness' about terminology, let's take a careful look at a specific example. Suppose you want to 
test $H_0: \mu = 100,$ vs $H_a: \mu > 100,$  where the population distribution is $\mathsf{Norm}(\mu, \sigma),$ with both $\mu$ and $\sigma$ unknown.
You randomly sample $n = 20$ observations $X_1, X_2, \dots, X_{20},$ obtaining
the sample mean $\bar X = 102.5$ to estimate the sample mean $\mu$ and sample population standard deviation $S = 13.2$ to estimate to estimate the population standard deviation $\sigma.$
The sample mean $102.5$ is larger than the hypothetical value $\mu_0 = 100,$
but that is not enough to reject $H_0$ in favor of $H_a.$ The issue is whether
$\bar X = 102.5$ is enough larger than $\mu_0 = 100$ to warrant rejecting $H_0.$
The $T$ statistic compares the difference $\bar X - \mu_0 =102.5 - 100 =2.5$ with the (estimated) standard error $S/\sqrt{n}= 13.2/\sqrt{20} =  2.95161.$
Specifically, 
$$T = \frac{\bar X = \mu_0}{S/\sqrt{n}} = \frac{2.5}{2.95161} = 0.847.$$
Testing with a fixed significance level and critical value: One way to test the hypothesis formally is to notice that $T \sim \mathsf{T}(\nu = 20-1 = 19)$ Student's t distribution with 19 degrees of freedom). assuming that $H_0$ is true. 
The critical value $c$ is set to cut probability $0.05 = 5\%$ (significance level $\alpha)$ from the right tail of this distribution. From a printed table of t distributions, or from software such as R, we find that $c = 1.728.$ We reject if $T = 0.847 \ge c = 1.728.$ Because this rejection criterion is not
met, we cannot reject $H_0.$ 
We say that there is not evidence in the sample to reject $H_0.$  By using the rejection criterion $T \ge c,$ we will reject
a true null hypothesis $H_0$ only 5% of the time we do such a test.
qt(.95, 19)
[1] 1.729133

One rejects $H_0$ for large values of the test statistic $T.$ In our example $T$ is not large enough. According to the figure we do not reject at level $\alpha = 5\%$ because the vertical blue line (observed $T$-value is not to the right of the vertical dotted red line (5% critical value).

Testing with a P-value: The P-value is the probability (assuming $H_0$ to be true), of a test statistic $T$ that is as extreme or more extreme (in the direction of the alternative), than the value observed.
Ordinarily, P-values are found using software. The t test procedure in most
statistical software will compute the P-value given the alternative hypothesis and the data. In our example, the P-value is the area $0.2038 > 0.5$ under the density curve of $\mathsf{T}(19)$ to the right of the vertical blue line.
1 - pt(0.847, 19)
[1] 0.2037689

One rejects the null hypothesis in favor of the alternative hypothesis if the P-value is smaller than 5%. So one rejects for small P-values. (If the vertical line for $T$ is far enough to the right, then the P-value will be small enough to reject $H_0).$
By contrast, if we had observed $T = 2.6 > c = 1.728,$ we would have rejected $H_0.$
In that case, the P-value would have been about $1.6\% < 5\%.$
 1 - pt(2.3, 19)
 [1] 0.01647629

