Maybe I misunderstood the question, but what you are describing sounds like a test-retest reliability study on your Q scores. You have a series of experts each going to assess a number of items or questions, at two occasions (presumably fixed in time). So, basically you can assess the temporal stability of the judgments by computing an intraclass correlation coefficient (ICC), which will give you an idea of the variance attributable to subjects in the variability of observed scores (or, in other words of the closeness of the observations on the same subject relative to the closeness of observations on different subjects).
The ICC may easily be obtained from a mixed-effect model describing the measurement $y_{ij}$ of subject $i$ on occasion $j$ as
$$
y_{ij}=\mu+u_i+\varepsilon_{ij},\quad \varepsilon\sim\mathcal{N}(0,\sigma^2)
$$
where $u_i$ is the difference between the overall mean and subject $i$'s mean measurement, and $\varepsilon_{ij}$ is the measurement error for subject $i$ on occasion $j$. Here, this is a random-effect model. Unlike a standard ANOVA with subjects as factor, we consider the $u_i$ as random (i.i.d.) effects, $u_i\sim\mathcal{N}(0,\tau^2)$, independent of the error terms. Each measurement differ from the overall mean $\mu$ by the sum of the two error terms, among which the $u_i$ is shared between occasion on the same subjects. The total variance is then $\tau^2+\sigma^2$ and the proportion of the total variance that is accounted for by the subjects is
$$
\rho=\frac{\tau^2}{\tau^2+\sigma^2}
$$
which is the ICC, or the reliability index from a psychometrical point of view.
Note that this reliability is sample-dependent (as it depends on the between-subject variance). Instead of the mixed-effects model, we could derive the same results from a two-way ANOVA (subjects + time, as factors) and the corresponding Mean Squares. You will find additional references in those related questions: Repeatability and measurement error from and between observers, and Inter-rater reliability for ordinal or interval data.
In R, you can use the icc() function from the psy package; the random intercept model described above corresponds to the "agreement" ICC, while incorporating the time effect as a fixed factor would yield the "consistency" ICC. You can also use the lmer() function from the lme4 package, or the lme() function from the nlme package. The latter has the advantage that you can easily obtain 95% CIs for the variance components (using the intervals() function). Dave Garson provided a nice overview (with SPSS illustrations) in Reliability Analysis, and Estimating Multilevel Models using SPSS, Stata, SAS, and R constitutes a useful tutorial, with applications in educational assessment. But the definitive reference is Shrout and Fleiss (1979), Intraclass Correlations: Uses in Assessing Rater Reliability, Psychological Bulletin, 86(2), 420-428.
I have also added an example R script on Githhub, that includes the ANOVA and mixed-effect approaches.
Also, should you add a constant value to all of the values taken at the second occasion, the Pearson correlation would remain identical (because it is based on deviations of the 1st and 2nd measurements from their respective means), whereas the reliability as computed through the random intercept model (or the agreement ICC) would decrease.
BTW, Cronbach's alpha is not very helpful in this case because it is merely a measure of the internal consistency (yet, another form of "reliability") of an unidimensional scale; it would have no meaning should it be computed on items underlying different constructs. Even if your questions survey a single domain, it's hard to imagine mixing the two series of measurements, and Cronbach's alpha should be computed on each set separately. Its associated 95% confidence interval (computed by bootstrap) should give an indication about the stability of the internal structure between the two test occasions.
As an example of applied work with ICC, I would suggest
Johnson, SR, Tomlinson, GA, Hawker,
GA, Granton, JT, Grosbein, HA, and
Feldman, BM (2010). A valid and
reliable belief elicitation method for
Bayesian priors. Journal of
Clinical Epidemiology, 63(4),
370-383.
