2
$\begingroup$

I am trying to figure out how to properly do regression analysis on a data set from a peer-review survey where individuals in the survey have an unequal number of responses. Below is description of my actual problem.

Consider a population consisting of $N$ individuals. You send a survey out to all $N$ individuals asking them to rank the quality of everyone (including themselves) in that population wrt some property based on a scale from NR (not rated) to 2,3,4,5. The NR responses are not counted. Suppose that $n$ of the $N$ individuals respond.

Some individuals in this population are of very high quality and so receive rankings from almost everyone in the population that respond (i.e. most of the $n$ responses are not NR). Conversely, some of the individuals are not of high quality and do not receive many rankings. For individual $i$, let $S_i$ denote the sum of all the rankings from the surveys received which are not NR and let $n_i$ denote the number of surveys received for individual $i$ which do not have NR rankings, from which we compute its overall score as $r_i$ = $S_i/n_i$.

Suppose you $only$ know $r_i$ for each individual and do $not$ have access to the original data (and so you do not know $n_i$ for each individual) and want to perform a regression analysis on the scores for some given set of covariates. My question is: how can one compensate or take into account the differences in the number of responses? Would weighted least squares naturally take into account that the individuals with more NR responses should "fit" the data better?

Thank you, Matt

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

First of all, this is a painful situation to be in, and if there's any chance you can get the original $n_i$s (or $i_i$s), then by all means pursue that before continuing. (You should also get the original number of people who received surveys, so you can distinguish missingness due to non-response from missingness due to choosing "NR".) You probably already knew that, but I figured it's worth being explicit.

It doesn't look you have any way to estimate the $n_i$s directly. You said you expect that people who get higher ratings on average will have been rated by more people, but that fact alone doesn't get us very far without an $n_i$ for anybody. So it looks like the most you can do is use a regression method that resists heteroscedasticity. Some suggestions can be found here and here.

Would weighted least squares naturally take into account that the individuals with more NR responses should "fit" the data better?

Not automatically. Weighted least squares requires specifying the weights (in this case, the $n_i$s).

Improve this answer
$\endgroup$
2
  • $\begingroup$ I appreciate your comments and concur with everything you've said. Unfortunately, the institution holding the data (which is U.S. News and World Report, FWIW) will not release the original data to me and mathematically, I can obtain the $n_i$ from the data itself. I actually did a trial run to try and see if the residuals showed evidence of heteroscedasticity (both by visual inspection and using White's test) and they did not, so my original idea will not apply to the data set with which I am working. Thanks again. $\endgroup$
    – Matt Brenneman
    Commented May 27, 2016 at 15:20
  • $\begingroup$ If you felt my answer was satisfactory, remember to accept it by clicking on the check mark under the voting arrows. $\endgroup$
    – Kodiologist
    Commented May 27, 2016 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.