I recall reading a story about a giant beverage company releasing a "new and improved" version of one of their beverages. They got positive feedback with surveys, but neglected to ask participants whether they preferred it to the old flavor, and it flopped when it went to market.
I'm in a similar scenario. I got a group of people to try out two versions of video learning software. One version had an extra feature enabled on the software. I asked them a six-question survey to measure whether it made the software more enjoyable to use.
My question is, what can I learn from this data? I graphed the frequency of responses of one of these features (below - 1 is "Strongly disagree" and 7 is "Strongly agree" that they found the feature useful/enjoyable), and the results shown are quite positive for this particular feature.
In a previous question a commenter suggested I use Cohen's Kappa to measure inter-rater reliability. If Cohen's Kappa is high enough to suggest good inter-rater reliability, and I show that the mean lies in the "Agree" range of the scale, can I then state that users found this feature useful? Is there any formal analysis that I can use for this?