2
$\begingroup$

I'm looking for a method to determine if the number of survey response received from a regional office are substantial enough to compare to each other year over year. For example, one year we might get 29 survey responses in a month from one site but only 5 responses for the same month the next year. Can I say those are truly comparable?

I guess what I am thinking of is similar to a Gallup poll or something like that, where they test to see if their response make up approximately the same mix of people between two time frames.

The reason we are trying to do this is to avoid a survey bias, as we know some regional sites consistently perform better than others, so if our sample contains more responses from the better performing sites, our results will be skewed favorably when they are really just due to having a better response rate (more observations) from certain cites.

Improve this question
$\endgroup$
4
  • $\begingroup$ This is not a question that can be answered with a "yes" or a "no". The more people you have, the more precise your estimates can be. But before we can say exactly how to do this, we'd need to know more about what you are testing. Is "performance" measured on a continuous scale? $\endgroup$
    – Peter Flom
    Commented Sep 28, 2013 at 14:32
  • $\begingroup$ Yes, I'd say its a continuous variable, because we measure it on a monthly basis however, we aren't really interested in performance this month compared to last month, but rather this month this year to the same month in the prior year. $\endgroup$
    – Eudora
    Commented Sep 28, 2013 at 15:21
  • $\begingroup$ @PeterFlom: Perhaps you understood the problem from the start. I was confused, but feel like the issue is very clearly spelled out now. (See my answer and the comments to it.) And I feel like I'm not qualified to address the problem beyond my opinions. Any further thoughts? $\endgroup$
    – Wayne
    Commented Sep 28, 2013 at 18:29
  • 1
    $\begingroup$ Bias is a function of the survey design, and it does not depend on sample size. In other words, an increase in sample size may not yield improvements in bias (documented in survey methodology literature). $\endgroup$
    – StasK
    Commented Mar 17, 2015 at 12:18

1 Answer 1

Reset to default
1
$\begingroup$

I've totally edited this answer based on your comments, below.

So what you have is technically called unbalanced panel data. You have a fixed set of sites that report daily productivity on a daily basis, where each site's data has missing days that are not reported. The number of missing days varies between sites, and within a site over time. Productivity is defined as the percentage of orders received and fulfilled that day.

Your suspicion is that sites will be more likely to report on days that are especially productive because they want to bump this fact up the chain. You want to calculate productivity on a monthly basis per region (collection of sites), comparing the same month across years, but you're worried that the mixture of sites that report a significant number of days in a particular month may change over time and thus not be representative of the region.

This went in a different direction that I originally thought, so I'm mainly clarifying so that someone like @Peter Flom might give better advice.

Two last questions from me: How many years of data do you have? Has the number of sites that exist, or the way they're grouped into regions changed over the time you've gathered data or has it been pretty static (except for the actual reporting, of course)?

Improve this answer
$\endgroup$
7
  • $\begingroup$ The responses are really a productivity metric, basically how many orders were completed versus how many were placed. The surveys are sent to all offices (sites), so we do know which office and which regions are responding. The response rate from the sites can be vary greatly though. We would obviously like to include as many sites as possible, but some sites will responds 29 times in one month, where another site might only respond 5 times in that month. $\endgroup$
    – Eudora
    Commented Sep 28, 2013 at 15:56
  • $\begingroup$ I guess what I am wondering is if it would be appropriate to check the distribution of the responses per site and possibly exclude some sites as outliers due to low responses (by each region I suppose). $\endgroup$
    – Eudora
    Commented Sep 28, 2013 at 15:59
  • $\begingroup$ This sounds more encouraging. Are the reports supposed to be daily and sometimes you actually get daily responses and other times it's more like the results from a whole week? Or are they supposed to be daily but sometimes they only report a single day in a week? $\endgroup$
    – Wayne
    Commented Sep 28, 2013 at 16:00
  • $\begingroup$ They are supposed to be daily and the report is only for a single day of work (never a week to catch up for failing to respond). So each survey received is one day for one site (which we group into our own regions as well, and that is the level we want to sample, that the site responses by region are consistent year over year). Unfortunately, we have no way of enforcing that the sites respond. Some sites respond fairly consistently, while other sites are pretty intermittent. $\endgroup$
    – Eudora
    Commented Sep 28, 2013 at 16:33
  • $\begingroup$ In the end, we want to make sure that the year over year sample contains approx. the same mix of sites (e.g. excluding sites that only responded a couple of times in one or the other of the samples). $\endgroup$
    – Eudora
    Commented Sep 28, 2013 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.