Suppose I have two drugs; A and B. B is a new drug that has to be introduced in the market.
$H_o\text{: Drug A and B are both equal}$
$H_a\text{: Drug B is better than Drug A}$
If I set my level of significance to let's say 5% $(\alpha=0.05)$. What does that really mean in terms of layman language?
In the common setup, you perform a test involving a test statistic, say $T$, and obtain a p-value, say $a$. Given that you have observed a value $t$ of the test statistic, the relation between $T$ and $a$ often is something like $$ a = P(T > t\ |\ H_0) \quad \text{or} \quad a = P(|T| > t\ |\ H_0) \quad \text{or} \quad a = P(T < t\ |\ H_0), $$ where $H_0$ is the null. The way of defining $a$ depends on the meaning of your statistic $T$, as it represents the probability of observing $T = t$ or worse given that the null hypothesis is true. It depends of what you judge to be a bad value of $T$ (evidence against $H_0$).
In your case, $T$ could be the difference in drug effect (bigger is better), i.e. $T = E_B - E_A$ and your p-value could then be $a = P(T > t)$. Observing a big value of $T$ gives more evidence in favor of the alternative hypothesis, that is against $H_0$.
That being said, if $a$ is the probability of observing what you have observed (or worse) during the experiment, obtaining a small value of $a$ means that what you have observed is unlikely to happen if $H_0$ is true, and so there is not much evidence for $H_0$. Thus, you usually reject $H_0$ in this case. The role of $a$ is to quantify how unlikely it is to observe $T=t$.
Now fix $t$ to be the value of the test statistic computed on the original experiement and say it corresponds to a p-value of $a=0.01$. Assuming that $H_0$ is indeed true, what would happen if you were to repeat this experiment over and over again? If $a$ is the probability of observing $T>t$, then this means you would indeed observe $T>t$ on approximately $0.01\times 100 = 1\%$ of you experiments. So even if $H_0$ is true, there is still a slight chance that you observe a small p-value. This "slight chance" is exactly what you specify as your significance level: you specify what it means for a p-value to be small (less than $5\%$ for example).
All this to say, it is intuitive to think of the significance level $\alpha$ as follows : if $H_0$ was true and you were to repeat the experiment many times, you would expect to (wrongly) reject the null hypothesis $(\alpha\times 100)\%$ of the times.
The level of significance, in an intuitive sense, describes how much evidence you need before making a claim of "statistical significance" for a finding. A p-value describes the likelihood of observing your data under the null hypothesis, which in this case, is that both drugs are the same. So, a p-value of 0.1 indicates that if both drugs really were the same, you'd see a difference as extreme as what you observed 10% of the time. A p-value of 0.01 indicates that you'd see the difference only 1% of the time if both drugs were the same.
The level of significance provides a decision threshold for p-values - either we consider a result "statistically significant", or we do not, and the level of significance is a pre-defined threshold for making that decision. This threshold must be set before doing any analysis, to avoid bias in moving the goalposts after the fact. A significance level of 0.05 is common, which means that we'll incorrectly reject the null hypothesis 5% of the time. If we want to be more sure that doesn't happen, we can set our significance level to something different up front, perhaps to 0.01 or 0.005, which will result in fewer spurious rejections of the null.
In terms that a doctor should understand, your alpha is the risk of a false positive, or in this case, claiming that drug B is better than drug A when there is in truth no difference.
