This topic, or at least this formulation of it, has proven surprisingly obscure, and the inconsistencies in the textbook haven't been much help. However, I think I have figured out the answer to most of the questions posed in the OP.

Definition of $\beta$

The text defines (experimental) $\beta$ as $\beta = \frac{P(X |S)}{P(X|N)}$, where $S$ denotes a signal trial, $N$ a noise trial. But this definition is of little use because $X$ is left undefined (see p. 29). What does $X$ mean here?

$X$ is the decision threshold that the operator (perhaps unknowingly) is using to categorize trials as signal or noise. Let $f_N$ denote the pdf of the stimulus during a noise trial, and $f_S$ denote the pdf of the stimulus during a signal trial. Then the formula is better rendered

$$\beta = \frac{f_S(X)}{f_N(X)}$$

which is sometimes described as "the ratio of the ordinate of the signal and noise distributions at the decision threshold." Here ordinate is an old-fashioned way of referring to the y-value or density of the pdf.

This definition assumes that both $f_N$ and $f_S$ describe normal variables of equal variance (but different means).

The text also defines $$\beta_\text{opt} = \frac{P(N)}{P(S)}$$ for the case where payoffs are unspecified, and then provides a more specific formula for optimal $\beta$ given a payoff matrix. What assumptions are made about the payoffs in the definition of $\beta_\text{opt}$ given above?

The longer formula is $$\beta_\text{opt} = \frac{P(N)}{P(S)} \frac{v_{CR} + v_{FA}}{v_H + v_M}$$ (where $v_{FA}$ and $v_{M}$ are typically negative). The two formulas are equivalent when $v_{CR} + v_{FA} = v_H + v_M$, e.g. when the payoff matrix is symmetrical.

Sluggish beta

If the optimal beta is high, then so is the optimal $X_C$, which corresponds to a conservative policy (i.e. averse to false alarms, per p. 29). So, I think the words in boldface should be switched.

I stand by this.

ROC curve and changes in signal probability

One practice question I have asks, what happens to $\beta$ when signal probability increases? (The question doesn't specify if we are talking about optimal $\beta$ or experimental $\beta$; instead it says "according to the ROC curve.")

I think this is a poorly formulated question.

• If the question is about optimal beta, then increasing the signal probability decreases $\beta_\text{opt}$, per the formula above.
• OTOH, suppose the question is about observed beta.
  • If the system is automated, then an increase in signal probability has no effect on the decision criterion, and $\beta$ is unchanged per the formula above.
  • If a human is making a decision, then in practical terms, humans tend to try to equalize the number of false alarms and misses, so increasing the signal probability will generate more misses, causing the user to adopt a more liberal policy, i.e. lowering beta.

(figure)

What point are the authors trying to make by including the signal probabilities $\begin{bmatrix}20 & 10\end{bmatrix}$ above this matrix?

I believe this is an attempt to show how optimal beta shifts in response to changing signal probabilities under the assumption that the payoff matrix is symmetrical.

What do the ROC curves shown above represent? Are they meant to represent idealized experimental data? Or some sort of generalization of the optimal $\beta$-value?

Basically, the latter. These are isosensitivity curves. In theory, a signal detection system (like a Bayesian classifier) with a fixed sensitivity—that is, one whose "internal" representation of the signal and noise variables creates two normal distributions of equal variance—can modulate its decision freely between a totally risky criterion ($\beta \to 0$) and a totally conservative one ($\beta \to \infty$), and doing so will create a curve with the form above, whose shape is determined entirely by the underlying sensitivity (the difference in the means of the normal distributions, called $d'$) and the assumption of normality and equal variance.

There is a nice interactive demonstration of this on this website. It provides the formula $$d'= \Phi^{−1}(p_H)−\Phi^{−1}(p_{FA})$$ from which the formula for the shape of the isosensitivity curve itself easily follows: $$p_H = \Phi\left( d' + \Phi^{−1}(p_{FA})\right)$$ where $\Phi$ is the cdf of a standard normal variable.

We can also get humans to adjust their internal criterion decision (albeit suboptimally, because of sluggish beta) in a signal detection task, e.g. by varying the signal probability or payoff matrix. However, because humans don't have a nice, normal internal representation of the stimulus to work with, doing so often results in a change in both beta and the sensitivity. The empirical sensitivity can be calculated by using the confusion matrix and applying the assumption of normal stimulus variables with equal variances.

(Formulas for this are given in the appendix to the chapter; the gist is that you figure out how far apart the means of $f_N$ and $f_S$ would have to be from $X_C$ in order to generate the confusion matrix in question.)