1
$\begingroup$

I need some help to find the correct method for my hypothesis. The aim is to find out whether the use of a certain procedure (case A: none / case B: use of my artifact) influences the time factor of a certain category (roles, competence, communication, deploy & release)   For this purpose, a questionnaire was sent out where users in my company had to assess both the current situation and a theoretical target situation. For example, one category had the following questions about the evaluation:

1) How do you assess the delivery and release process currently practiced in your area with regard to delivery reliability? (A higher value is better) 2) How do you assess the deployment and release process described in the model with regard to delivery reliability? (A higher value is better

The participants were able to rate from 1 (insufficient) to 8 (very good) on a rating. First i thought this can be an ordinal scale, but i think its more of an interval scale.   The goal is now to test my assumptions that everything is better, more amazing, faster with the new model (artifact) 😊   Since I am not that fit in statistics, I have the following questions: In my opinion I try to check whether there is a homogeneity problem or not, am I correct? I use the exact Fisher test, but as you can see in the question topic, i am not sure about it?   Here are is one of my hypotheses:

H0: The use of the developed scaling framework and the number of unplanned work in the competence category are independent.

H1: The use of the developed scaling framework and the number of unplanned tasks in the competence category are not independent.

in the contingency table you can see the rating 1-8, as well the two models called (ohne Artefakt and Mit Artefakt) enter image description here

$\endgroup$

1 Answer 1

0
$\begingroup$

Treating responses as nominal. Treating your Likert scale of responses 1 though 8 as categorical data: at the 5% level of significance, a chi-squared test marginally rejects (P-value 0.048) the null hypothesis that responses for 'mit' and 'ohne' have the same distributions of Likert values.

ohne = c(9,11,16,11,13,13,13,10)
mit  = c(5 ,6, 5,12,12,27,18,11)
TBL = rbind(ohne, mit)
out = chisq.test(TBL); out

        Pearson's Chi-squared test

data:  TBL
X-squared = 14.213, df = 7, p-value = 0.04752

round(out$resi, 3)
       [,1]   [,2]   [,3]   [,4]   [,5]   [,6]   [,7]   [,8]
ohne  0.756  0.857  1.697 -0.147  0.141 -1.565 -0.635 -0.154
mit  -0.756 -0.857 -1.697  0.147 -0.141  1.565  0.635  0.154

The largest Pearson residual $1.565$ points to Likert category 6, with more observed responses $27$ for 'mit' than would be expected if Likert scores were independent of whether or not your 'procedure' is used.

The chi-squared statistic $14.2$ is the sum of $2(8)=16$ contributions $C_{ij}=\frac{(X_{ij}-E_{ij})^2}{E_{ij}},$ where $X_{ij}$ are observed counts and $E_{ij}$ are expected counts based on the null hypothesis. Pearson residuals are signed square roots of the $C_{ij}.$

[The Fisher exact test for TBL in R simulates a P-value. I did not run this test. But you have enough data for a valid chi-squared test.]

Treating responses as interval data. You suggest treating Likert scores as interval data. There is considerable controversy whether this is a valid assumptions. However, taking the data to be interval data, with sample sizes as large as 96, it seems OK to use a 2-sample Welch t test.

x = rep(1:8, ohne)
y = rep(1:8, mit)
t.test(x,y)

        Welch Two Sample t-test

data:  x and y
t = -2.7712, df = 186.92, p-value = 0.00615
alternative hypothesis: 
  true difference in means is not equal to 0
95 percent confidence interval:
 -1.4087369 -0.2370964
sample estimates:
mean of x mean of y 
 4.552083  5.375000 

The average Likert score $(\bar Y = 5.375)$ for 'mit' respondents is significantly higher than the average score $(\bar X = 4.552)$ at the 1% level of significance. [The Welch test does not assume population variances to be equal.]

Note: Once assuming Likert scores as interval, the justification for doing a t test with such data is that sample means of $n = 96$ observation are nearly normally distributed---even though the scores 1 through 8 are not.

For example: Re-sampling with replacement groups of 96 scores from among mit shows that sample averages are roughly normal.

set.seed(520)
a = replicate(10^4, mean(sample(mit, 96, rep=T)))
hist(a, prob=T, col="skyblue2", ylim=c(0,.6))
  curve(dnorm(x, mean(a), sd(a)), add=T, col="red")

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.