How can one define the power of a randomized clinical trial?

The statistical power of a study is a measure of the ability of a randomized clinical trial (RCT) to detect a difference statistically significant of the treatment effect in the treatment arms.

1. How can one define the power of RCT?

For the logistic regression (i.e. with a binary outcome), the treatment effects were usually reported in terms of Odds Ratio (OR).

logit(p) = log(OR) = log (p/(1-p))
where p is the probability of presence of the
characteristic of interest.
1. Is there a mathematical relationship between the power of a RCT and OR? And how to interpret or understand this sentence: "80% power to detect a treatment vs. control OR of 0.8"?

1. The definition of (statistical) power is more than the "ability" of a trial to detect a statistically significant difference. It is the actual probability of the trial to reject the null hypothesis. If your question is actually how does one calculate the statistical power, it usually boils down to a few idealized assumptions. For a logistic model with a single binary covariate, it is the sample size in either arm, the baseline frequency of the event, and the odds ratio. There's a complicated effect size statistic due to Cohen that is an algebraic combination of these values with which one can look up power. However, a simple power calculator like G*Power is a really useful tool to explore and learn.

2. Binary analyses are typically driven by the number of events (assuming "events" are rarer, like with cancer). So in one design the control group may have a 20% event rate, and the exposed group has a 40% event rate, whereas in another design the control group has a 5% event rate and the exposed group has a 12% event rate; these designs may have the same OR of effect, but the power for the second design is at least half. In general, the OR won't tell you what you need to know about power. If you are reviewing prior literature where the OR was reported, you have to do some careful thinking about the design and sampling frame to try and construct some other plausible values for your prospective study.

Also, I'll note that in your statement:

$$\text{logit(p)} \ne \log(OR)$$

$$\text{logit}(p)$$ is the fitted value from the logistic regression. The difference of two $$\text{logit(p)}$$s is a log odds ratio. In the earlier example,

$$\text{logit}(0.20) = -1.386$$, $$\text{logit}(0.40) = -.405$$, and $$\text{logit}(0.05) = -2.944$$, $$\text{logit}(0.12) = -1.992$$.

So the log odds ratio is 0.981 and 0.952 in both cases, i.e. very close.