I have a survey where I ask people their ratings for something. If they vote 0-4, I rank them as "con", 5-7 as "on the fence," and 8-10 as "pro."

So now I want to split up all my data by region, such as their state.

However, some states have a lot more replies than others.

I can't tell which of these results are significant and which aren't, or if there's anything else I need to take into account. For example some states simply have a lot more people.

Is there a simple way to determine which results in this context are meaningful and which are not?


Here are some things to consider.

1) Why are you splitting up your outcome variable into coarsing groupings? You lose variance by doing so. That may be theoretically fine, or after some modeling you may see that it is appropriate to do so. But modeling 3 levels with either a linear model (probably not appropriate, but could be after model checks) or an ordinal/multinomial logit/probit will likely give you different results than modeling the outcome as continuous. I wouldn't jump to the conclusion of breaking down the data in that way before doing some testing unless my theory was really strong; I bring this up because you haven't provided any details on why you want to do this.

2) It is relatively simple to account for differences in states - just include them as covariates. Population, for example, can be included as a covariate (consider entering it in logarithms rather than raw). This will adjust your results by each state's population. However, if this is survey data and you have sampling weights, those weights may account for that, in which case including population as a covariate would have a different interpretation.

3) To model state differences you could enter them as fixed effects (K-1 dummy variables, where K is the number of states). Alternatively, you could consider a random coefficient model, where the state is a normally distributed random effect. There are tests for this that are important to use. Without knowing more about what you are modeling or what you have done, I won't get into the details too much here.

You need to provide more context to your statement "I can't tell which results are significant." What results? Did you model it? How? What are you looking at that is confusing?

  • $\begingroup$ I appreciate the response, but honestly, replies like this really discourage me from stuff like this because I am totally unfamiliar with 95% of the things you reference in your post. All I'm trying to do is say "Okay, for Washington state, we see that x number of people fall into the Con category, y number of people fall into the On-The-Fence category, and z people fall into the Pro category. Are these results significant / accurately representative, or do we need more data to be sure?" $\endgroup$ Dec 12 '14 at 15:23
  • 1
    $\begingroup$ Sorry, I didn't mean to sound curt. It would help if you put some examples of the output you are looking at and said a little more about your example. You haven't given enough info to answer your question or provide guidance that gets to your exact point. $\endgroup$ Dec 12 '14 at 15:29
  • $\begingroup$ Maybe I misunderstand but I assumed that there was a cutoff in statistics for which we can claim something is statistically significant. I am basically trying to determine at what point can we say "we have enough data for these results to be meaningful" in terms of output. $\endgroup$ Dec 12 '14 at 15:36
  • $\begingroup$ Robin, your efforts to answer the question are very much appreciated. But since you're relatively new here, please do not be offended if I remind you that when the original poster "hasn't given enough info to answer the question," then unless you are really sure you understand the question anyway, you should first consider responding with a comment requesting clarification rather than posting a full answer. $\endgroup$
    – whuber
    Dec 12 '14 at 16:20
  • $\begingroup$ I disagree that I have not given enough information, though. What I am asking about is a very common concept in statistics, but I don't know how to properly apply it here. $\endgroup$ Dec 12 '14 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.