In the definition of the Fisher Information matrix: $$ \begin{align} I(\theta)_{ij} &= \mathbb{E}_{x \sim p(x \,;\, \theta)}\left[ \left(\frac{\partial}{\partial \theta_i} \log p(x \,;\, \theta) \right) \left(\frac{\partial}{\partial \theta_j} \log p(x \,;\, \theta) \right) \right] \end{align} $$ is the expectation on the RHS computed using the same value for "$\theta$" that is passed into $I(\theta)$ ?
Or is the density "$p(x \,;\, \cdot)$" in the expectation computed using the true but unknown parameter value of the parameter, call it $\theta^*$, so that: $$ \begin{align} I\left(\bar{\theta}\right)_{ij} &= {\large \int} \left( \frac{\partial}{\partial \theta_i} \log p(x \,;\, \theta)~\Biggr|_{ \theta=\bar{\theta}} \right) \left( \frac{\partial}{\partial \theta_j} \log p(x \,;\, \theta) ~\Biggr|_{ \theta=\bar{\theta}} \right) \, p(x \,;\, \theta^*) \, dx \end{align} $$
In this second definition, the Fisher information matrix would tell us how much information the true distribution (as specified by $\theta^*$) provides about the value of theta at location $\bar{\theta}$.
Alternatively, if the same value for theta (namely $\bar{\theta}$) is also used in the density "$p(x \; \cdot)$", then the meaning of the Fisher information matrix is something like "how much information does the density specified by $\bar{\theta}$ contain about itself?" And I'm not really sure how that quantity would be useful in practice.