What is "one" in leave-one-out cross validation Lets say I have $x_{ij} \sim Bernoulli(p_j)$ and $d_j = \sum_{i=1}^n x_{ij} \sim Binomial(n, p_j)$ I could do binomial logistic regression by regressing the logit of $p_j$s on some predictors. If I was then interested in doing some cross validation like LOOCV, what is the difference in leaving out one $x_{ij}$, computing the $d_j$s, and fitting, vs leaving out one of the $d_j$s?
Maybe more concretely:
Consider data
library(tidyverse)
set.seed(123)
df <- expand.grid(p=1:10, rep=1:12) %>% 
  mutate(x = rbinom(10*12,1,prob=plogis(-2 + 0.3*p)))

df_grouped <- df %>% group_by(p) %>% summarize(d = sum(x), n=length(x))

I can then either do
glm(x ~ p, data=df, family="binomial")

or
glm(cbind(d,n-d) ~ p, data=df_grouped, family="binomial")

Now if I am interested in doing leave one out cross validation what is the difference in leaving out a row in df, fitting the model, and computing statistics (repeat for all rows) VS leaving out a row in df_grouped, fitting the model, and computing statistics?
 A: Here's what I think. First consideration: The answer should depend on what you actually want to predict. If the $x_{ij}$ (or rather their associated predictors) come in one by one and you're interested in the prediction quality you get for every single one of these, using all individual observations you already have, leave out one $x_{ij}$ row at a time. If a whole bunch of observations (or a $d_j$) at the same predictor position (let me call that $z_j$) comes in at once and you want to predict that, leave out a $(z_j,d_j)$. In fact I can also imagine an intermediate situation where, say, $d_j$ comes in for a set of, say, 10 $x_{ij}$ at a time, but you may already have other observations at the same $z_j$ (the way you formulated it it looked like $d_j$ referred to all observations with the same $z_j$, but this is not necessary in my view), then I'd leave out one "observational set" at a time.
The difference between these is that if you have, say, 100 individual $x_{ij}$ in sets of 10 each, obviously in the first version you have 99 observations to predict the 100th one, in the second version you have only 90 observations to predict a $d_j$, and if the $z_j$ are all different for different $d_j$, you won't have any observation at the same $z_j$ (although as I said, this isn't necessarily always so), so chances are you'll be slightly worse off than in the first version.
Second consideration: When doing the actual prediction of new data, you will use all 100 observations that you already have in the case above, and it is actually equivalent predicting a new $x_{ij}$ by $\hat x_{ij}$, say, or $d_j$ by $n\hat x_{ij}$. As the first LOO-CV version predicts from 99 observations and the second one from 90, the first one will have a smaller bias (it might have a larger variance but I don't think that'll trump the bias here, I may be wrong though). For this reason arguably you could use version 1 for both cases.
On the other hand, if indeed you want to predict $d_j$ and all $d_j$ indeed come with different, i.e., new $z_j$, it may be more realistic to use the second LOO-CV version in which no observations with the same $z_j$ are used for prediction (it may improve prediction quality to have such observations already, and if this is not the real situation, that may lead to an optimistic estimate of the prediction error).
If your aim is comparing models rather than assessing the final prediction error, I expect hardly any difference between the two approaches (the second version may be problematic if numbers of observations are critically low).
