How can I know a random small subsample has a good performance?

Question

Sorry, if the title is confusing. not native speaker here.

I ran a survey study which had 15 7-point likert scale questions. Each question is ask people to judge how creativity an object in the picture. I received 153 responses in total.

What I try to understand is whether those people who answered the survey did a similar judgement as experts do.

I have 15 experts completed the same survey and compute the mean rating for each picture. For the 153 responses (I call this group as crowd), I also calculated the mean rating for each picture.

Here is the mean rating data of the two groups (E-experts, C-crowd)

E   C
3.36    2.33
1.43    1.53
1.07    1.19
2.43    2.69
2.14    1.92
4.64    4.22
4.71    4.36
4.00    3.24
4.21    3.20
5.07    4.78
4.36    3.41
4.64    5.02
6.14    5.50
5.79    5.73
6.00    5.95

my first question about the data is whether the two group did the judgment similar? could I do a simple t-test comparing the mean rating of 15 questions and tell the result?

second, since I do the survey for an application design. In the real system, for each picture , I could expect only 15-20 people come to rate it. So I want to know if 153 responses I collected now say they do similar judgements as those experts(n=15) do, whether I could say random 15 people could also do similar judgement?

How can I know a small sample (a subsample) has a similar performance as a big sample has?

one way I think about is to random subsample 15 people from the 153 responses for 10000times, and use the mean rating to be compared with the mean rating of 153 responses. but I'm not sure whether it is a right way to do it.

Please do ask for more information if you get confused by my language.

Answer 1

Since no one else has replied, let me give it a shot. As to your first question, the t-test you describe would tell you if the means for E and C were different overall, but you would be getting your error estimate from among the 15 questions rather than from among your many respondents. That is not ideal. You could consider doing a t-test between C and E for each of your 15 questions. This would let you get your error estimate from among your respondents. You would have to address the issue of inflation of alpha due to repeated t-tests. Perhaps you could do one 2x15 ANOVA (that is, E/C by question) with repeated measures rather than 15 t-tests. Additionally, a correlation between the 15 pairs of means would tell you if there was a similar pattern for your E and C groups, overall. Sometimes that is as important as absolute agreement. Also, have you looked at the level of agreement within your expert group? As to your second question, my guess is that an N of 15 or 20 is not likely to give you much power. Have you given thought as to what amount of disagreement between E and C is a meaningful amount of disagreement? If so, you might be able to calculate power for various N's. Assuming power is low with N's of 15 to 20, perhaps you might adopt another approach. You might think of tabulating the data and then collecting a new larger data set if the E-C difference is greater than an amount you chose.