Regression Using Surveys with Unequal Number of Responses I am trying to figure out how to properly do regression analysis on a data set from a peer-review survey where individuals in the survey have an unequal number of responses. Below is description of my actual problem.
Consider a population consisting of $N$ individuals. You send a survey out to all $N$ individuals asking them to rank the quality of everyone (including themselves) in that population wrt some property based on a scale from NR (not rated) to 2,3,4,5. The NR responses are not counted. Suppose that $n$ of the $N$ individuals respond.
Some individuals in this population are of very high quality and so receive rankings from almost everyone in the population that respond (i.e. most of the $n$ responses are not NR). Conversely, some of the individuals are not of high quality and do not receive many rankings. For individual $i$, let $S_i$ denote the sum of all the rankings from the surveys received which are not NR and let $n_i$ denote the number of surveys received for individual $i$ which do not have NR rankings, from which we compute its overall score as $r_i$ = $S_i/n_i$.
Suppose you $only$ know $r_i$ for each individual and do $not$ have access to the original data (and so you do not know $n_i$ for each individual) and want to perform a regression analysis on the scores for some given set of covariates. My question is: how can one compensate or take into account the differences in the number of responses? Would weighted least squares naturally take into account that the individuals with more NR responses should "fit" the data better?
Thank you,
Matt
 A: First of all, this is a painful situation to be in, and if there's any chance you can get the original $n_i$s (or $i_i$s), then by all means pursue that before continuing. (You should also get the original number of people who received surveys, so you can distinguish missingness due to non-response from missingness due to choosing "NR".) You probably already knew that, but I figured it's worth being explicit.
It doesn't look you have any way to estimate the $n_i$s directly. You said you expect that people who get higher ratings on average will have been rated by more people, but that fact alone doesn't get us very far without an $n_i$ for anybody. So it looks like the most you can do is use a regression method that resists heteroscedasticity. Some suggestions can be found here and here.

Would weighted least squares naturally take into account that the individuals with more NR responses should "fit" the data better?

Not automatically. Weighted least squares requires specifying the weights (in this case, the $n_i$s).
