What is the $p$ in Bernoulli distribution? In the Bayesian theory of probability, probability is our expression of knowledge about a certain thing, not a property of that thing. However, I always see people treat $p$ as a parameter that needs to be estimated. They set up a prior for $p$, usually in the form of a beta function and then update it as "realizations" of this variable come in.
Even the great bayesian Jaynes sometimes gives the impression that he is "estimating the probabilities" or looking for the $p$ which best "fitst the data":

Now we wish to take into account only the hypotheses belonging to the
  ‘Bernoulli class’ $B_m$ in which there are $m$ possible results at
  each trial and the probabilities of the $A_k$ on successive
  repetitions of the experiment are considered independent and
  stationary;

Probability Theory, E. T. Jaynes, page 297
This makes me confused, because $p$ is not a probability, since it is a property of the random variable and it is not a frequency, since the variable represents a single event.
 A: $p$ is a parameter which specifies the "success probability", for which we have prior and posterior probability distributions. 
We may for example have a coin for which we are not sure if it is fair ($p=0.5$) or not ($p\neq 0.5$). Even so, fairness, or lack thereof, is a property of the coin. We just happen to be unsure about that property of the coin.
We then, for example, specify a beta prior distribution as a prior probability distribution over the possible success probabilites in $[0,1]$. That prior may, for example, be inspired by looking at the coin, assessing if it "looks" fair. If it looks fair, we will be inclined to specify a prior with lots of probability mass around $p=0.5$. 
In other cases, say, when forming a prior belief about the probability with which a football player will be successful at his next penalty - also a Bernoulli outcome, either a goal or not - we will be inclined to put more probability mass on $p$ around 0.8, because professional football players score on most penalties.
We then toss the coin/observe the player a couple of times, and summarize the information in the likelihood function, to obtain the update, i.e., the posterior.
A: For a random variable $X \sim \operatorname{Bernoulli}(p)$ defined on a probability space $(\Omega, \mathcal{F}, P)$, the parameter $p$ (a number) is the probability of a certain event, namely, the event $\{X = 1\}$.
That is,
$$
p = P(X = 1).
$$
The single number $p$ completely determines the distribution of $X$ since for any Borel set $B \subseteq \mathbb{R}$ we have
$$
\begin{aligned}
P(X \in B)
&= \mathbf{1}_B(0)P(X = 0) + \mathbf{1}_B(1) P(X = 1) \\
&= (1 - p) \mathbf{1}_B(0) + p \mathbf{1}_B(1).
\end{aligned}
$$
(Here $\mathbf{1}_B$ is the indicator function of $B$.)
This is why the family of Bernoulli distributions is parametereized by the interval $[0, 1]$.
This fact is independent of a frequentist or Bayesian interpretation of statistics: it is just a fact of probability.
If we're being Bayesians, then we'd want the parameter $p$ to be a random variable itself with some prior distribution.
Formally, we can say that our parameter is a random variable $\Pi$ supported on $[0, 1]$ and we have
$$
X \mid \Pi \sim \operatorname{Bernoulli}(\Pi),
$$
which means that
$$
\begin{aligned}
P(X = 1 \mid \Pi) &= \Pi, &
P(X = 0 \mid \Pi) &= 1 - \Pi
\end{aligned}
$$
almost surely (or
$$
\begin{aligned}
P(X = 1 \mid \Pi = p) &= p, &
P(X = 0 \mid \Pi = p) &= 1 - p
\end{aligned}
$$
for any $p \in [0, 1]$).
In this case, the parameter $\Pi$ (a random variable) is the conditional probability of the event $\{X = 1\}$ given $\Pi$. This conditional probability, together with the distribution of $\Pi$ (the prior distribution), completely determines the distribution of $X$ since$$
\begin{aligned}
P(X \in B)
&= E[P(X \in B \mid \Pi)] \\
&= E[\mathbf{1}_B(0)P(X = 0 \mid \Pi) + \mathbf{1}_B(1) P(X = 1 \mid \Pi)] \\
&= E[(1 - \Pi) \mathbf{1}_B(0) + \Pi \mathbf{1}_B(1)] \\
&= (1 - E[\Pi]) \mathbf{1}_B(0) + E[\Pi] \mathbf{1}_B(1)
\end{aligned}
$$
for any Borel set $B \subseteq \mathbb{R}$.
In any case, frequentist or Bayesian, the usual parameter of Bernoulli data is the probability (either marginal or conditional) of some event.
A: 
In the Bayesian theory of probability, probability is our expression
  of knowledge about a certain thing, not a property of that thing.
  However, I always see people treat $p$ as a parameter that needs to be
  estimated. They set up a prior for $p$, usually in the form of a beta
  function and then update it as "realizations" of this variable come
  in.

This is irrelevant. It has nothing to do with interpreting the meaning of probability, since this is not about philosophy, but about well defined mathematical object. You see people discussing estimating value of $p$ because you look into statistics handbooks and statistics is about estimating things, but $p$ is a parameter of distribution, it can be known, or unknown.
If $X$ is a Bernoulli random variable with probability of "success" $p$, then $\Pr(X=1) = p$ by definition. So $p$ is a parameter of this distribution, but it is also probability of "success".

This makes me confused, because $p$ is not a probability, since it
  is a property of the random variable and it is not a frequency,
  since the variable represents a single event.

Yes, random variable describes some "single event", so if you are going to toss a coin, the possible outcome is a random variable because it is uncertain. After you tossed the coin and know the outcome, it is no more random, the outcome is certain. As about probability, in frequentist setting you consider hypothetical scenario where you would repeat the coin tossing experiment huge number of times and the probability would be equal to the proportion of heads among those repetitions. In subjective, Bayesian setting, the probability is a measure of how much do you believe that you will observe heads.
The above is however irrelevant to question what $p$ is. It is a parameter that is also equal to probability of "success". The question how do you interpret the probability and what does it mean is a different question.
