0
$\begingroup$

I have very little (i.e. high school level) stats training, so forgive me if anything in here doesn't make sense.

A team at my work has performed a training exercise for about 250 employees and has conducted a survey afterwards about how the training has affected their day-to-day work. The survey comprises about 10 statements about the training, each with 6 possible responses (Strongly Disagree", Disagree, Slightly Disagree, Slightly Agree, Agree, and Strongly Agree).

I have been tasked with figuring out how effective different aspects of the training were, based on the survey results.

The initial approach suggested was to count the percentage of Agree or Strongly Agree answers for each question and use this as an effectiveness measure.

However, I've been thinking that if somebody "strongly" agrees or disagrees with a given statement, that their opinion should be weighted more heavily (since they have formed a definite opinion on the subject), and if somebody only "slightly" agrees or disagrees, their opinion should not be as heavily weighted (since they are pretty neutral about the subject).

So, for instance, each "slightly" answer should be counted as 0.5 of a response and each "strongly" answer should be counted as 1.5 responses.

Is there any precedence for this sort of analysis, or am I just overcomplicating things?

$\endgroup$
2
$\begingroup$

The trouble with weighting is that your results will be arbitrary. For example, if you organize your responses on a 1-6 scale (1 being strongly disagree and 6 being strongly agree), then you're saying that the "distance" between a 1 and a 2 is the same as the "distance" between a 2 and a 3. (Here I use "distance" to indicate difference or the gap between what one number represents and another number represents.)

What I would suggest is, depending on your analysis, looking into an ordered model of some sort. The ordering indicates that StD < D < SlD < SlA < A < StA, but doesn't specify how large the distance is between any two options. I prefer an "ordered logit model" and that should suffice for your analysis if you have a large enough sample (which it appears that you do). This will also let you see how other factors affect their response on the survey, if you have that sort of information (i.e. gender, time with the company, department, etc.) available.

In broad strokes, ordered logit is going to be a fancy regression method that works with categorical data that has an order but isn't necessarily equally spaced out. Regression (as you may remember) is a way to measure the association between two variables by saying if one variable changes by $X$ amount, we expect the other variable to change by $Y$ amount. (I know you haven't taken a stats class recently, so hopefully this elucidates some of the ideas. There should be information online about how to conduct this sort of analysis.)

$\endgroup$
  • $\begingroup$ Doing a bit of googling, this sounds like what I want. But I am having trouble trying to figure out how to implement it in Excel. Any tips? $\endgroup$ – Lexo Oct 26 '15 at 17:07
  • 1
    $\begingroup$ A quick Google search turned this up: real-statistics.com/multinomial-ordinal-logistic-regression/… I generally implement it in R, so I'm far more familiar with that and can't provide much help in Excel. Hope this helps, though! $\endgroup$ – Matt Brems Oct 26 '15 at 17:17
  • $\begingroup$ Thanks for the link :-) Would I do this for each question in the survey, or is there a way to do it for the survey overall? $\endgroup$ – Lexo Oct 26 '15 at 21:14
  • 2
    $\begingroup$ You would want to do this for each question in the survey, as each question in the survey will have different values. If an individual puts "Strongly Disagree" for one question and "Slightly Agree" on the next, the results wouldn't make much sense - especially if the questions vary between positive and negative. $\endgroup$ – Matt Brems Oct 26 '15 at 21:17
  • 2
    $\begingroup$ The point I'm trying to make is that it is inappropriate to weight it because when you weight these, you give them arbitrary values. For example, if you gave "Strongly Disagree" a 1, "Disagree" a 2, and so on up to "Strongly Agree" getting a 6, you are saying that the "distance" between a "Strongly Disagree" and a "Disagree" is the same as the "distance" between a "Slightly Disagree" and a "Slightly Agree." (Here I use "distance" to indicate difference or the gap between what one number represents and another number represents.) Any weight you make is arbitrary and should be avoided. $\endgroup$ – Matt Brems Oct 28 '15 at 18:21
1
$\begingroup$

One easy to do analysis available from many "off the shelf" stats packages would be a factor analysis. Factor analysis is a technique that has been around since the turn of the last century and was initially used in analyzing the results from intelligence tests. It's kind of like building a regression model but without a dependent variable insofar as it pushes correlated content -- the attributes in your survey -- together based on their association with an underlying, unobservable dimension called a "factor." The factors represent statistical combinations of the attributes and are typically meaningful and directly interpretable. For instance, based on an FA of IQ test items, you can easily imagine two broad constructs emerging: one dimension for verbal and a second for quantitative intelligence, skills or ability.

The key metric to analyze would be the "loadings" or correlations between the dimensions and the specific attributes in your survey. These "loadings" would provide a rank ordering of the attributes' importance. As such, weighting would be an automatic byproduct of the methodology.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.