Meaning of Standard Deviation when scales vary On crowd sourcing sites like mechanical turk, we may ask people to rate two models in the scale of, say 1 to 5.
Obviously, some people will give 5 for the good one and 1 for the bad one, while some may give 3 and 2, etc. In other words, each person has their own scale within the specified range.
As such, standard deviation turns out to be large.
Let's say we ended up with ratings for two models, collected from 5000 people where each people rated both models. We have
average 3.35 with std 1.0 for model A, and
average 3.17 with std 1.08 for model B.
Since the ratings for A and B significantly overlap when applying avg+/-std for the models, I am not sure whether A's superiority is statistically significant.
So my question is, could such results as described above be considered significant despite large std? If so, should other metrics like p-value be introduced to make the significance more clear?
===========edit======
I am now grouping the results into three cases, where A was rated better than B, equal to B, or worse than B.
How would the situation change?
 A: Presuming you are referring to standard null hypothesis statistical testing then I would say you must calculate a p-value as this is what you compare to your alpha criterion to determine if a statistically significant difference exists. The simplest parametric test of the null hypothesis that Model A = Model B would be a t-test. T-tests are relatively robust to differences in standard errors. Welch's t-test, or the unequal variances t-test, proceeds from the assumption that your standard errors do in fact differ.
However, as @EdM mentions in the comments, you may want to consider different analyses that are better suited to the data. Typically a rating score like the one you describe is on an ordinal scale. Because ordinal data has may not have equal intervals between the ratings, it is not reasonable to perform multiplication and division on those values, as is required to calculate a mean or standard error. Your scale would need to be interval or ratio to carry out those calculations (see: http://psychology.okstate.edu/faculty/jgrice/psyc3214/Stevens_FourScales_1946.pdf). Instead, you might want to consider a test that can be meaningfully applied to ranks such at the Mann-Whitney U-test.
Having said that, there are those that advocate for a t-test in these cases despite the violation of its assumptions, noting that performance is often very similar, e.g.
De Winter, J. C., & Dodou, D. (2010). Five-point Likert items: t test versus Mann-Whitney-Wilcoxon. Practical Assessment, Research & Evaluation, 15(11), 1-12.
Zimmerman DW, Zumbo BN. 1993
. Rank transformations and the power of the Student t -test and Welch t ′-test for non-normal populations. Can J Exp Psychol
  47
: 523
–39. 
Edit: I note from your comment that your data is paired (i.e., you measure each individual twice). You would want to ensure to use a test that accounts for that relationship, a paired samples t-test if you go the parametric route or the Wilcoxon Signed Rank test for non-parametric.
