Welcome to the site, Brad. If I were in your situation, I would use a mixed (a.k.a. random) effects model to analyze your data. These models easily handle uneven group sizes and have no problem with singleton clusters (with just 1 member) provided that such clusters are (ideally) outnumbered by non-singleton clusters.
In a 2-level mixed effects model,
$$y_{ij} = \beta_0 + u_{0j} + \epsilon_{ij}$$
the outcome $y_{ij}$ is a linear function of the overall intercept $\beta_0$, between-cluster error $u_{0j}$ (the random intercept), and within-cluster error $\epsilon_{ij}$ (the residual). The model thus splits variation in $y_{ij}$ into two error components, each associated with their own variance terms $-$ $\theta$ for within-cluster variation and $\psi$ for between-cluster variation.
Because singleton clusters have 0 within-cluster variation they do not contribute to the model-estimated $\theta$ nor to the determination of how variance is partitioned across $\psi$ and $\theta$. However, they do contribute to the estimation of any slopes ($\beta$) and the estimation of $\psi$ + $\theta$, which is the total variance in the outcome.
Another reason to favor mixed effects models over "fixed" effects models comes into play when you want to predict each cluster's value on the outcome (call it $\tilde{y_j}$). In a fixed effect model a singleton cluster's $\tilde{y_j}$ is completely based on information from that single individual, which is why these models are sometimes referred to as "no-pooling" models. In contrast, the mixed effects model assigns a empirical prior distribution to the random intercepts $u_{0j}$:
$$u_{0j} \sim N(0, \psi)$$
The mean of this normal distribution is 0, which is understood in relation to the intercept ($\beta_0$) from your model with variance $\psi$ to be estimated by the model. Using empirical Bayes prediction, a singleton cluster's $\widetilde u^{EB}_{0j}$ is pulled back toward the overall intercept $\beta_0$. These models are sometimes called "partial pooling" models for these reasons. The degree of partial pooling is based not only on the within-cluster sample size, but other factors as well. See this answer for information on the empirical Bayes correction factor that helps determine how much pooling occurs.
