Calculate inter-rater noise using Kahnemans (2021) approach

I need help calculating signal and noise based on the method described by Kahneman et al. (2021) in their book "Noise." They provide a technique for quantifying noise between raters assessing the same cases. Imagine five psychologists (raters) assessing the level of depression for 10 patients (cases) Here's the setup: Let's say you have raters as rows and cases as columns. With zero noise, every rater agrees completely, so all variance is between cases (signal). Kahneman et al. further divide noise into (simplifying a bit):

• rater noise: stabile differences between raters, e.g. different means for different people
• residual noise: idiosyncratic noise that’s not explained by anything.

To calculate noise, they recommend:

1. Calculate the mean of the SD from each column (the sd of each case) (Total noise)
2. Calculate the SD of the means from each row (the means of each rater) (Rater noise)

You can then calculate residual noise by 1 - 2 = Residual noise. My challenge is to calculate both noise and signal from a dataset. Additionally, I want to report noise as the intraclass correlation coefficient (ICC), but I am struggling to replicate Kahneman's method using other decomposition methods. For instance, using a random effects model with lme4::lmer() gives me different results.

I have simulated some data to illustrate my unsuccessful attempts. I am also questioning whether Kahneman et al.'s approach to analyzing noise is advisable, as I cannot find any other research utilizing this method. Lastly, I need a sensible way to measuring and reporting uncertainty. Is there a way to bootstrap my way to a confidence interval here?

# Load necessary packages
library(lme4)
library(tidyverse)
library(performance)

# Set seed for reproducibility
set.seed(1234)

# Simulation parameters
n_raters <- 600
n_cases <- 30
n_obs <- n_raters * n_cases

# Simulate random effects
rater_effects <- rnorm(n_raters, mean = 0, sd = 1)  # rater random effects
case_effects <- rnorm(n_cases, mean = 0, sd = 2)    # case random effects

# Simulate dat
dat <- expand_grid(rater = factor(1:n_raters), case = factor(1:n_cases))
dat$$ratereffect <- rater_effects[as.numeric(dat$$rater)]
dat$$caseeffect <- case_effects[as.numeric(dat$$case)]
dat$$residual <- rnorm(n_obs, mean = 0, sd = 3) # Residual noise dat$$y <- dat$$ratereffect + dat$$caseeffect + dat$residual dat # Fit mixed-effects model (# This approximately retrives the parameters from the simulation) model <- lmer(y ~ 1 + (1|rater) + (1|case), dat = dat) # Display summary of the model summary(model) performance::icc(model,by_group = TRUE) # Noice decomposition # rater noise = 7.9 % # case variance (signal) = 25 % # residual noise = 100-(7.9+25) = 67.1 % # Decomposition of noise only # rater = 7.9/(7.9+67.1) = 10.5 % # residual noise = 100 - 10.5 = 89.5 % # Kahneman, Sibony, Cass (2021) calculation dat <- dat |> mutate(case = paste0("case",case)) |> select(rater,case,y) |> pivot_wider(values_from = y,names_from = case) # calculate the mean of each row (rater) dat <- dat |> select(-rater) |> rowwise() |> mutate(row_mean = mean(c_across(everything()))) # take the sd of row means sd_ratermean <- sd(dat$row_mean)

# take the standard deviation of each case
sd_case <- dat |>
pivot_longer(everything()) |>
summarise(sd_case = sd(value),.by = name) |>
pull(sd_case)

# take the mean of case sd's
mean_case <- mean(sd_case)

total_noise <- mean_case
level_noise <- sd_ratermean
pattern_noise <- total_noise-level_noise

level_noise_percent <- level_noise/total_noise
pattern_noise_percent <- pattern_noise/total_noise

cat("level noise =",round(level_noise_percent,2),
"\npattern noise =",round(pattern_noise_percent,2))

$$$$

• Are these methods meant to give the same answer? In any case, I think there is an error in your computation of the total, level and pattern noise: total_noise and level_noise are (on the scale) of std. deviations. So you need to square them to do the decomposition: pattern_noise^2 = total_noise^2 - level_noise^2. Commented Jun 15 at 14:30
• Im not sure what I expect. What do you think of the Kahneman way of calculating noise? Based on your comment it might be sensible, but should squaring total noise and level noise. When I square, I get a percentage that's very close to the ICC calculation. Commented Jun 17 at 9:43
• Is it fair to say that the Kahneman approach is erroneous, since it does the percentage calculations with standard deviations (non-additive) instead of variance (additive)? Commented Jun 17 at 10:02
• Not sure your statement is correct. I checked out the kindle version from the library & I'm looking at Chapter 6, The Analysis of Noise. In the section "Judges differ: pattern noise" the formula is System Noise^2 = Level Noise^2 + Pattern Noise^2`. It would be strange to do the decomposition on the std. dev. scale. Commented Jun 17 at 10:45
• I think you are correct. The formula does indeed say to square each component. In my defence, Kahenman et al dont mention this step when going through the judges example (see page 74), which is the only calculated example in the book. Commented Jun 18 at 7:08