1
$\begingroup$

I faced a challenge recently with our commercial team about our annual market survey result. As you can guess, our commercial team has no background in statistic or math etc.

A brief introduction, after careful sampling design based on population and stratified probability sample from national phone database bank, we selected randomly phone numbers and interviewed 10,000 people. The stratas are specified by Geography (18 regions) and within stratas the phone numbers were selected randomly within strata quota requirements were implemented by interlacing Age (10-19; 20-29; 30-39; 40-49; 50-59; 60-69, 69+) and Gender (Males, Females). We asked people if they listen/stream podcasts or stream/watch film on Netflix/HBO and if yes how often (daily, weekly, monthly etc). I weighted the survey based on official population data and produced a report.

One of our competitors run a similar survey with 12,000 samples (as far as I am aware) with a similar method via phone. Our result does not match and it came as no surprise to me. For example, we are saying 10% of persons aged 10 and older listening to a podcast or watch video on Netflix in a typical month but our competitor's report says 14% listen or watch daily. or our result shows 59% of the population watch/stream at least once a month, but their survey says 46% etc.

And this became such a headache for me, our sales team keeps questioning why we report lower or higher compared to company X survey. I tried to explain them two different surveys from two different companies will not give the exact same result, As we are sampling different samples and we should expect a change on the results as this is claim based. also explain there is a sampling error in each survey. For example if the reality of gender distribution is 50%, when I sampled and ask people about their age, there is a X% sampling error. well, It didn't work since I guess they cannot follow my explanation.

So I was wondering if anyone can help me to better explain or formulate the why question for folk who has no idea how survey and sampling work

$\endgroup$

1 Answer 1

1
$\begingroup$

What are the actual counts behind those percentages? Maybe presenting counts will make things easier to grasp. Here are my very rough thoughts (I have no experience with this sort of surveys):

You have 18 regions, 7 age groups, and 2 genders: 252 groups. If you survey 10000 people (is that in total?), you have only ~40 people per group. Now, 59% of 40 is 24 people watching once a month vs 18 reported by the competitors (46% of 40, assuming the same sample size, take or leave). The difference 24 vs 18 is not that impressive and if you consider that you compare tens of different groups (if not 252) then it's easy to see that such differences are expected.

$\endgroup$
1
  • $\begingroup$ Thanks @dariober yes, 10,000 is total interviews. and for example for 10% daily user (after weighting ) , actual count unweighted is 1084 (1084 out of 10,000 claimed are daily user) but I donot have actual count for competitors version $\endgroup$
    – DanielG
    Dec 11, 2020 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.