Statistical difference between groups on number of 'negative' answers I have survey responses on whether several (~50) different factors (such as the work-culture) have a positive, neutral, of negative influence on how people perceive their job. I would like to test if there is a significant difference between different groups (e.g. men-women, people who have a children-people who don't have children, etc.) on how 'negative' they are on each factor. In other words: have, for instance, women answered significantly more negative than men when presented with the factor of work-culture? What statistical test should I use for this analysis?
And do I perform the correct statistical test on the % of negative answers or on the number of answers (together with the N of each group)?
Thanks!
 A: There are several ways to approach this, depending on precisely what you want to know. Based on the following objective

I would like to test if there is a significant difference between different groups [...] on how 'negative' they are on each factor.

I would propose the following:

*

*Consider two groups you wish to compare - for example male and female employees. Call these groups A and B.

*Determine whether responses in either group are more negative than in the other.

Your data (negative, neutral, positive) are ordinal, meaning they have an ordering ('negative' is more negative than 'neutral' and 'positive', and 'neutral' is more negative than 'positive').
You can then use the Mann-Whitney U-test to determine responses from groups A and B are significantly more negative than the other. You can map the three responses to -1, 0, and 1, and run the test like so (the code below is python 3.7).
from scipy import stats


A = [1, 0, 0, -1, -1, -1, -1, 0, -1, 0, 1, -1, 1, 0, 0, -1, 0, 0, 0, 1]
B = [1, 1, 0, 1, 1, -1, 0, 1, 1, 1, -1, 0, -1, 1, 0, 1, 1, 0, 1, 1]


res = stats.mannwhitneyu(A, B)
# Prints 'Test results: p=0.016 (U=116.0)'
print(f"Test results: p={res.pvalue:.3f} (U={res.statistic})")

Note that when running 50 such tests, it's very likely that some will be significant purely by chance, so you might want to use a more conservative significance threshold than .05.
Also note that this only gives you a measure of how certain you can be that the responses differ. If you want to say something about how much they differ, you should consider the fraction of people in groups A and B who respond negatively, neutrally, and positively.
A: If you have access to a computer, try the bootstrap.

*

*From the real responses of the group of e.g. men, draw a simulated set of responses with replacement. Call this "a bootstrap replication" of men's responses. Do the same for women.


*Repeat the previous step very many times. For each of these pairs of bootstrap replications, subtract the proportion of responses of interest from the men's replication from the proportion of responses from the women's replication.


*In the step above, you will get a long list of differences. Sort this list from smallest to largest. If the middle, say, 90 % of values exclude 0, you might have an effect worth investigating!
The neat thing about the bootstrap is that you don't have to know a whole lot of theory to do it.
Practically, something like this:
import math
import random

# P = positive, I = indifferent, N = negative
workculture_men = 'PPPPINNPPPNPIIIINPPPNIIIPNPI'
workculture_women = 'NPPPIIIPPPPINNNNNPPNPPPNNPPPNNNPPPIINPNPIIPN'

def bootstrap_proportion_negative(responses):
    replication = random.choices(responses, k=len(responses))
    prop_neg = sum(r == 'N' for r in replication)/len(replication)
    return prop_neg


def bootstrap_difference(men, women):
    for i in range(1, 500):
        yield bootstrap_proportion_negative(men) - bootstrap_proportion_negative(women)


distribution = list(sorted(bootstrap_difference(workculture_men, workculture_women)))

lower5percent = distribution[math.floor(0.05*len(distribution))]
upper5percent =distribution[math.ceil(0.95*len(distribution))]
print(f'Middle 90 %: [{lower5percent:.2f}, {upper5percent:.2f}]')


