I think we can interpret the problem in the following sense:

If we keep the masked tokens all masks that means the corresponding Y would be independent of the mask because the mask is the only choice and hence the independence is satisfied. No matter what the X is the Y remains the same distribution. And when we randomly choose the X that also satisfied, meaning that for any word the distribution of the target would be the same.

The only way we can break the independence is we bias the representation towards the true words, thus the correlation/association/dependence holds.

If the actual words are observed with a probability, say 10%, the distribution Y would be much different from if they are not observed(either all masks or random sampling).

I think we can interpret the problem in the following sense:

If we keep the masked tokens all masks that means the corresponding Y would be independent of the mask because the mask is the only choice and hence the independence is satisfied. No matter what the X is the Y remains the same distribution. And when we randomly choose the X that also satisfied.

The only way we can break the independence is we bias the representation towards the true words, thus the causation holds.

If the actual words are observed the distribution Y would be much different from if they are not observed(either all masks or random sampling).

I think we can interpret the problem in the following sense:

If we keep the masked tokens all masks that means the corresponding Y would be independent of the mask because the mask is the only choice and hence the independence is satisfied. No matter what the X is the Y remains the same distribution. And when we randomly choose the X that also satisfied.

The only way we can break the independence is we bias the representation towards the true words, thus the causation holds.

If the actual words are observed the distribution Y would be much different from if they are not observed(either all masks or random sampling).

I think we can interpret the problem in the following sense:

If we keep the masked tokens all masks that means the corresponding Y would be independent of the mask because the mask is the only choice and hence the independence is satisfied. No matter what the X is the Y remains the same distribution. And when we randomly choose the X that also satisfied, meaning that for any word the distribution of the target would be the same.

The only way we can break the independence is we bias the representation towards the true words, thus the correlation/association/dependence holds.

If the actual words are observed with a probability, say 10%, the distribution Y would be much different from if they are not observed(either all masks or random sampling).